The Roger Pirates were constantly fighting the Holy Knights, who were trying to get Shanks back. But every time, they were defeated by the Roger Pirates. Gaban also confirmed that none of the Holy Knights(they faced) possessed Conqueror's Haki.
@Elder Lee Hung ,
@Sentinel ,
@DarkestKnightofSpoilers ,
@Red Admiral
Now the big info everyone asked for:
Gaban and Shanks fight Harald, but they are powerless because Harald possesses Conqueror's Haki, that is far too strong for them(They can´t deal anything, Harald CoC simple protects himself similar to Xebec, all they can do is to see how Harald destroys everything).
Harald also slaughters all the giants in the castle during the fight!!