I agree with the admirals being restricted and Fujitora acting. But sharing the same rank and authority, I doubt Aramaki would've picked a fight with Fujitora, just like Fujitora didn't pick a fight with Aramaki for trying to enforce slavery on those same people. We know the admirals clashed with the Revos, but haven't seen Aramaki clashing with Fujitora. Being pissed at Fujitora doesn't mean he will actually pick a fight with his equal in military rank.
And again, why does Kaido and Shanks have such an easy time obliterating Yonko top commanders using a small portion of their strength?