If that was meant to be what's inferred, he would have said, "I just want the freedom to do whatever I want."
If you take a closer look at Mihawk's character, his situation is the personification of living peacefully. He spent his time on a secluded Island, planting crops and drinking, without having to worry about being bothered by anyone. He was even annoyed that Perona ended up on his island, because what did that take away from him? His peaceful life of not being bothered by anyone or anything. Then when he does feel like he is starting to get bored, he heads out to sea to engage in something to kill time, before making his way back to his island to resume his quiet, peaceful life.
It's honestly weird to think that Mihawk is just bored 24/7 because there is no one he can swordfight with that is on his level. How depressed of an individual he must be if the ONLY possible way on the entire planet for him to not be bored is to have a challenging Swordfight.
Just because he likes to go out and engage in combat when he does get bored, doesn't mean that he's just wanders around life in this suspended state of boredom.