Sure, perhaps I exaggerated. I wasn't specifically referring to you though. I wrote that while having in mind what other people had said in other occasions as well.
Also, my point is that it's quite uncertain why exactly he fainted while talking to Osome; It wasn't directly stated that it was mainly because of one thing or because of another thing, so it's open for interpretation.
Considering how this is also something that happens in real life, realistically speaking, I would say that that he fainted because of a sense of relief he had as he talked to a pretty woman knowing that he had been able to fulfil his duty which was to beat Queen, so all the accumulated fatigue that wasn't noticeable while he was pumped up in his fight kicked in all of a sudden making his body rest as it knew that it could rest. So it's not like he necessarily wouldn't be able to keep fighting and using Ifrit Jambe and invisible speed if he really had to after beating Queen and facing a new enemy immediately afterwards; It's just that the sense of relief allowed him to rest.
And what caused the fatigue wasn't specifically the invisible speed, but all the things he did in Onigashima, including beating all those people while running around with Luffy and being chased by Black Maria, besides while carrying Zoro, and also all the damage he took from Black Maria and from Queen, all summing up, of course, with the effort he had to make to move at invisible speed and to use Ifrit Jambe and Diable Jambe so many times.
So yes, invisible speed is certainly one factor that made him lose stamina, but not the only one, nor necessarily a very relevant factor.