My personal guess
rayleigh and shakky have hancock
They see the world government hunt down roger's family and decide to leave her with the amazonians, they have active bounties, so they decide to leave her in amazon lily.
They remain in contact from time to time which is where the familiarity comes from
but they didnt 'raise' her.
Now you might ask why its not similar with gaban
well it kind of is
elbaf is an isolated kingdom.
difference is gaban can reside there , amazon lily cant have men around.