When we were told that ¨Ryuma slayed a dragon over the flower capitol¨? Not Ryuma slayed a ¨Devil fruit user who had Dragon powers¨?
We also know the story of Ryuma pretty much? It´s based on a past manga Oda made, where he clearly killed an actual Dragon? Unless you can somehow explain that despite giants living past a century, with *generations* of giants not having been able to get the fruit and beat Ragnir, why some random Giant, within the last 800 years would have eaten the fruit, gone to Wano(?) and no future giants would care, lol?