Yes, people correctly pointed out that using some random sticks with nails or pipes is just a sign of people/kids in poverty trying to weaponize anything they can get their hands on, and child Aramaki looks like some orphan living in the streets, unlike Issho who looks like he comes from a noble family enjoying formal education and training at a young age.
You don't need to be exposed to swordsmanship from a very young age to turn out as a proficient swordsman, in Rayleigh's flashback, young Roger didn't appear with a blade either. Similarly, Coby as a kid was a weakling without any combat ability but later became a proficient martial artists knowing how to use Rokushiki and even swordsmanship.