He didn't hold any grudge against Fuji for example, infact they seem to be on good terms.
Becuz akainu aren't aware of the fact how he let luffy go again. Sengoku is aware of that not akainu.
Also not allowing him to enter marine base is grudge in itself. I really don't think even the highest authority should have the power to restrict someone entry in their base to someone who's just below them not like any vice admirals. If fujitora declared his alliance with Revo or SH. While telling what Marines were doing to Warlords. I'm pretty sure akainu was coming for Fuji ass too.
Not to forget even gorosei questioned akainu about aokiji leaving like that.