I think the speaker's intent and beliefs about the meanings of the words they're using is relevant. Context is everything in language.
I think the issue that sort of overlaps what you are talking about is sometimes when people assert things "aren't offensive", what they really mean is that they think they should be free to use derogatory language without others being upset.
I think the issue that sort of overlaps what you are talking about is sometimes when people assert things "aren't offensive", what they really mean is that they think they should be free to use derogatory language without others being upset.