Well yeah, if Ryuma had a brother, you wouldn't be a direct descendant of Ryuma from his brother, but you'd just be a descendant of his bloodline.
However, if Ryuma had two kids, and those two kids had two kids, and those kids had two kids, and so on and so forth, you'd have a multitude of people who would be Ryuma's direct descendant.
So it's not like being a direct descendant of Ryuma is limited to one straight path down the tree.