I certainly agree that it is a double standard. I am reminded of a debate I had with my sister many years ago. I asked her if a man hit a woman, was she justified in hitting him back? Yes. But if a woman hits a man, is he justified in hitting her back? She said no. I thought that was ridiculous, but then she made a good point: the difference is that the potential for harm is generally much greater when a man hits a woman, and that problems stemming from female-on-male violence is less prevalent and socially damaging than male-on-female violence.
Obviously there are exceptions, but I think the same can be said about this. No matter how many times a black guy calls a white guy a cracker, it won't change the power differential between the two. But every single time a white man calls a black man a ethnic group member, he is endorsing a particular worldview that still affects and harms a lot of people.
I understand both sides of this argument, I just think that the potential harm caused by being called a cracker is infinitely less than being called a ethnic group member or a Christkiller or whatever.