I guess that the chains are made of a super reinforced metal, if not from
kairōseki itself. Even a DFless person like Garp is powerless to do anything, as
kairōseki is seemingly indestructible by any known means (not even Zoro could cut it back in Enies Lobby, when he already was able to cut steel).
It's the same principle of why Jinbe or other strong DFless prisoners couldn't just break their chains in Impel Down, it's supposed to hold even the strongest people in the world.