Yes fans shout, there was another bunch of dickheads behind singing 'Blackburn's a shithole' through the streets. That will antagonise locals and we have the same at the CBS. Inside the ground it's fine and banter. Outside it's a dickhead thing to do.
But, you're missing the point completely, most will ignore him. I'm sure if you travel regularly you'll have seen people started on for a lot less and if he's on his own, we know cowards will often target him rather than other group I mentioned, so ask yourself when you say it's harmless, would you personally jump in to defend him and risk your own banning order if someone started on him?