When Kaido wanted to go to Marineford Shanks walked up to him, and that was the end of it.
When Akainu and Blackbeard wanted to let loose at Marineford Shanks walked up to them, and that was the end of it.
When Mihawk walked up to Shanks, Shanks asked him if he's looking for a fight, and Mihawk wanted none of it.
When Shanks walked up to Marineford and said he will take on any challenge, Mihawk walked away from it.
Only Whitebeard tried to have a go at Shanks, and their clash split the sky in half. Mihawk having a go at Whitebeard got cut short by Jozu, and he had to settle with a duel against Vista.
That's how these people get to keep these titles. When Shanks shows up, they don't want no beef