People who are uncomfortable with you are often very uncomfortable with themselves.

It is not only your right but your duty to defend yourself from near mindless attackers.   Your best defense is always words said dirrectly to the attacker.   At least call them on their act if not anything.