Fighting =/ being easily defeated
Fighting =/ attacking
His bounty matters because it should reflect his achievements. Less than 40 million in two years is nothing, he went against the emperors.
And let's not forget how ruthless the Kid pirates are against civilians.
Hawkins stated that they
fought until both Kidd and Killer couldn't move
Invasion in BM territory required some fighting, otherwise Killer would be the best sneaker/infiltrator in OP (Kidd fought a Commander, is not far fetched to think that Killer fought someone too). If you're implying so, I'd take it without a question.
Red Hair Pirates are Red Hair Pirates, the most balanced Yonko crew, progresses don't come just by winning anyway
And bounty reflects achievement
known by the WG/Navy/Journalists, Caribou words imply that a lot of fake stories are told about Kidd Pirates (BM territory invasion is an example)
Zoro did the second effortlessly. They are air slashes with no CoA, not a big deal. BM lightings are just lightings, they shouldn't be stronger than Enel's ones. And I'm not talking about his best moves. Everybody tanked those in Skypiea
Kaido's slashes are different than CP9 slashes, I'm pretty sure about it. Killer was deflecting multiple slashes while running through them, is worth to be mentioned.
Same goes for Big Mom's lightnings.
Killer being a veteran level character doesn't make him weak, it's extraordinary he reached such a level with no training and in just two years.
Killer faces Commanders, not Veterans