First one doesn't show up for some of us
The supposed contradiction is that he didn't give a shit about Luffy, then showed up with a grudge against Luffy or the Straw Hat crew in general. And you say he let Luffy pass. But no. He wanted to focus on finding Yamato and beating Sasaki, and it's compared to that that chasing Luffy wasn't worth it. Again, he straight up says he wants to kill someone immediately after (Shanks). He wants to become an All Star and it's only my guess that it somehow plays into his goal, but yeah, we know his goal isn't Luffy.
He doesn't pick a fight, he encounters Jinbe because Jinbe was already there fighting his men, and says oops, Luffy got away. He didn't let Luffy pass. And then he goes on to explain that it's a sore spot but his grudge is with Shanks, and Jinbe plays in both because of his statement right there and ALSO because of Jinbe's past ties into his own.
Simple: he wanted Yamato and didn't really care about Luffy. He caught Jinbe fighting his men and explained his grudge against Shanks. The grudge spreads to Luffy and to Jinbe because of his declaration, but he also had interest in Jinbe because of the Nika thing.
Oddly enough, if you claim that he ignores Luffy and then let Luffy pass later, that doesn't sound like a contradiction either.