Words I munched on all weekend from Norm Gerras:
Racist incidents are, indeed, not just a matter of what is intended, as I have argued many times on this blog. But neither is the 'perception of other people' an adequate criterion. If it were, anyone could claim any word or gesture to be racist and that would settle the issue of whether it was. You could not have a flimsy or unfounded accusation of racism. There has to be some basis in the history and symbolism of the particular racism being alleged - whether anti-black, anti-Jewish, anti-Romani, anti-Chinese, or whatever - linking the incident or utterance in question with that 'tradition' of racial prejudice and persecution.
Gerras' definition makes me feel uneasy because history and traditions (or rather precisely perceptions and interpretations of them) can be and are often wrong/problematic.