The caves on the opposite hill, Khandagiri, can be reached either by the long flights of steps leading from the road, just up from the main entrance to the Udaigiri caves, or bu cutting directly across from Hathi Gumpha via steps that drop down from cave 17. The latter route brings you out at caves 1 and 2, known as the "Parrot Caves" for the carvings of birds on their doorway-arches. Cave 2, excavated in the first century BC, is the larger and more interesting. On the back wall of one of its cells, a few faint lines in red brahmini script are thought to have been scrawled 2000 years ago by a monk practising his handwriting. The reliefs in cave 3, the Ananta Gumpha or "Snake Cave"- serpents decorate the doorways - contain the best of the sculpture on Khandagiri hill, albeit badly vandalzed in places.

Caves 7 and 8 both house reliefs of tirthankaras on their walls as well as Hindu deities which had, by the time conversion work was done, become part of the Jain pantheon. The best place to wind up a visit to Khandagiri is the modern Jain Temple at the top of the hill. Aside from some old tirthankars in the shrine room, the building itself, erected during the nineteenth century on the site of a much earlier structure.