I think this has a more simple explanation that we might expect. People who eat Devil Fruits can't swim.
Imagine trying to pull a fucking Giant out of the ocean? Yikes, that dude is just dead. It's already hard enough to carry giants on a ship, a Giant with a Devil Fruit would be such a weakness at sea.
There is that one dude who's shield ate a Devil Fruit, though.
NGL, Goldberg is kind of a baller.