No because they are both jokes with negative percpetion attached to them that people use to cope about whatever character they don't like not REALLY being a top tier. That is all.
Shanks is not a "haki-man for example, he is a swordsman. Arguably the only "haki-men" are Garp and I guess Koby? But nobody would ever call them such because Haki is literally basic shit you just NEED to be able to use it in combat at least somewhat if you want to be even a vague threat overall.
Devil fruit merchants at least sorta exist in the form of characters like Sugar for example. Characters with either zero haki, or such little haki that they are utterly reliant on their fruits instead.
Most character don't fit either definition.
Like, let us take Imu for example. They are upwards of 800 years+ old. If they are NOT a devil and instead eat a devil fruit? Unless they are an abysmal match for their fruit Imu is bound to have mastered their devil fruit better than literally any other fruit user on the planet.
BUT, the same is true of their Haki as well. Imu reigns supreme as the ruler of the world. That is Top 1 willpower aka Haki set up there. Imu also for sure has Haki since he one shot Sabo, the logia user, with one tail/arrow attack. So unless Imu just learned Haki like last week ago they more than likely trained Haki for 800 year+ as well maming them a haki god. Thus?
If you took Imus fruit away? They still have Haki. If Oda reveals a haki off-switch/will suppression of some kind? They still got the fruit power. They are not overly reliant on either, and mastered both, thus are neither a fruit merchant nor haki-man/woman/being etc