There are a lot of assumption here.
Also what about Renly and Stanis? They weren't top tiers or even big.
Renly was 6’2 and Stannis was 6’5 lol, they were both big guys. Renley in particular was said to look identical to a Robert. Sansa described Renly as the most beautiful man she had ever seen in GoT. Renly is, as far as I can tell, the shortest Baratheon we’ve ever heard of at 6’2. Lol
You have to gauge their actions in life/portrayal to gauge them as fighters. Rogar Baratheon was said to be a huge man who was muscled like a bull, weilded a double bladed axe into battle, was well respected across the realm for his prowess in battle, and dreamed of facing Maegor Targaryen in 1v1 combat. He boasted openly that he wanted to do this when Maegor was alive, and Rogar was an enemy of Maegor since he sheltered Jahaerys who Maegor wanted dead. Rogar literally in life did not fear Maegor Targaryen, one of the only lords of his day who openly rebelled against Maegor with zero fear.
Rogar’s younger brother (who Jahaerys killed) had personally fought in the Third Dornish War, and had many victories against rival groups until Jahaerys finally brought Vermithor into the battle. I feel like his relationship to his hugely strong and Chadlike brother, as well as his recorded history of victory in battle would definitely suggest that this guy was no joke, and Jahaerys killed him 1v1.
You’re right that simply being born a Baratheon doesn’t mean someone is a top tier or even a warrior, you have to look at their deeds and actions during life, and yes genetics do matter when we have no physical descriptions of someone, and this guy was a warrior Baratheon with a history of victory in battle lol. I highly, highly doubt the guy was anything less than a great fighter.