We know that Loki’s fruit is suited for a giant to eat
Likewise I think other races could have fruits specifically suited for them. In particular, I think the yami yami no mi is the fruit of the buccaneers.
The yami gives the user extra pain. Buccaneers have shown the greatest tolerance to pain of all races. Kuma was able to power through his own death in order to save Bonney.
If mastery over pain is the key to mastering the yami, it is something only a buccaneer can do as buccaneers can endure limitless pain via the power of love.