This explains everything; the Celestial Dragons are the only people more racist than Zoro in this series, so, of course Zoro's opponent is a Celestial Dragon, or a former one in Mihawk.
Mihawk and Shanks became rivals, possibly due to Mihawk being ordered to hunt down Garling's child and retrieve them, and eventually friends, with Shanks making Mihawk realize how pathetic the life he lived is.
This is why Mihawk lives in total isolation, on a deserted island (besides the Humandrills). His island, Kuraigana (Gloomy) Island, was once home to a kingdom known as the Shikkearu Kingdom, which fell. From the Wiki:
Nine years before the current storyline, a terrible war was fought on this island. The Humandrills inhabiting the forest, gifted with the capability of learning from humans, watched the conflict, learned how to fight using weapons, and became fearsome fighters. At some point after the war's end, Mihawk arrived on this island and made the castle his home.
Either Mihawk participated in a Celestial Dragon Manhunt on Kuraigana, and feels remorse over it and lives alone in penance, or Mihawk obtained the island by intervening in a Celestial Dragon Manhunt, IMO.
If the war was really that damaging, why would the castle be kept safe? Because, it wasn't a war; the Celestial Dragons easily took over, and when Mihawk showed up, they quickly retreated, leaving him the island.
This is why the giant cross is there; it was cut by Mihawk to serve as a grave for those murdered on the island.
We can tell it was created by Mihawk thanks to your spooky waifu, Perona, in the anime (providing you count the Time Skip filler as real).