Idk....
But what's clear is that it indeed hinted during Croco conversation with Mihawk that he has a great trust issues.
Mihawk backstories is unknown except rivalry with Shanks. We'll find out more........maybe
After Roger's dead the people who sits at the top of pirate world are the remnants of Rocks pirates - Kaido, BM, WB and Whangzhi( probably) and then Shanks came in power and take him down , this what i believe.
The Rocks remnants rules the sea after Roger dead for a long time, they are at the top and many other strong pirates challenge them but got defeated. This can be easily tell just by headcanon.
Some Warlords are among the victims
Though Mihawk is strange one. Let's see if he got history with other OG Yonko, but i doubt since it will likely only be with Shanks that he has history.....