It's just like how black people can call each other the n word
they usually never mean it, they just know that they're capable of doing it and that it won't offend the other if it's coming from them, so they say it.
But have some random white guy call them the hard R and they're gonna slap the shit out of them.
Don't insult someone you don't know, they're not gonna take it nearly as easily as they would if it was from someone that they know and trust, because they'd rightfully think you're being serious.
Also, the opinions do matter. The main reason why homophobia is banned is because it's highly toxic and judgmental of homosexuals, and it can hurt them.