This chapter shouldn't be taken too seriously in terms of power levels.
The whole battle is depicted like a game more like an actual conflict, in my opinion. First of all, it only took three nights and days, which is low for an off-panel clash even by high tiers standards (check Jack versus the dukes and Jinbe versus Ace), let alone when top tiers are involved, and none of the participants seemed to suffer important injuries. Especially, the assumed two strongest after Roger and Whitebeard, Rayleigh and Gaban, were untouched. Also, it is obvious that they're just friendly rivals, as shown by the relationship between the captains and how they all end up exchanging gifts. An all-out battle in which they're trying to kill each other seems next to impossible, and therefore it is absurd to take this in a serious way; even when there are real motifs for Whitebeard to reassure his place the conflict doesn't scalate in the slightest (he's obviously angry about Roger taking Oden, to the point of being the only moment he arguably used his fruit considering the good state of the island).
And by the way, I can't understand how some people infere that Rayleigh and Gaban would be defeated by Oden because of Roger's words. Funny face on him plus Rayleigh stressing he's making up an excuse because he wants to face him himself already shows that he was just joking. Let alone taking Roger's attack as proof of Oden's power level, implying that Roger would even try to badly hurt a guy he has been wanting to meet for four years and may be important for reaching Laugh Tale.