The holiness of Kashi, aka Varanasi, is perhaps related to its geography. In Kashi, the river Ganga, which normally runs from the Himalayas to the sea in the south, takes a turn and moves northwards. This reverse flow of the Ganga is considered spiritually significant because it is believed that one can manifest certain tantric ideas in such places.
In tantra, all our bodily fluids move in a downward direction. We eat food which is transformed into flesh, blood, and bone, and eventually, it is secreted in the form of sweat, semen, and menstrual fluid, which move southwards. Our head always represents the north. However, it is the call of the sages, who perform tapasya or spiritual austerities, to retain these fluids and make them move in the reverse direction, towards the head. This grants them spiritual powers known as Siddhi, allowing them to be clairvoyant, to control space and time, walk on water and fly in the air. So perhaps the northern movement of the Ganga represents this shift, making this a geographically potent zone, which ancient sages found sacred enough to build a city on. According to local lore, gods loved the city of Kashi when they came to earth.
Once, the city suffered a drought and Brahma went around looking for someone to end it. He found Ripunjay, who said to end the drought, he would perform the necessary Vedic rituals and Dharmik practices. His only condition was that all the gods should return to their specific abodes and leave him alone in Kashi to conduct his rituals. This was accepted by the gods, who left for their respective domains, and Ripunjaya performed the rituals that brought rain back to the land. He established Dharma, in Kashi, and became a great king. He performed the Ashvamedha Yagya with 10 horses, a highly unusual feat, which gave rise to the Dasashwamedh Ghat. It became a prosperous and happy kingdom. Ripunjaya was renamed King Divodas, a name that occurs in the Rigveda.