What does being lonely at the top, have to do with wanting to live in peace?
And there is plenty of strong people he could fight if he really wanted to. They might not be Swordsmen, but if it's conflict he seeks, there is no shortage of candidates.
He wants a good sword fight. Not just a fight against a strong opponent. But in the same vain, he also wants to live a peaceful life.