Primebeard ruled in front of the throne and was labelled as the King of the seas, it was Whitebeard's era after Roger's death. He wasn't interested in the One Piece but ruled the era. While Mihawk said it would be too much stress to be engaged in Yonko business. Imagine Primebeard riding on Buggy's clown ship and being distressed at the thought of being peer to the other Yonko.