Don't feel bad for not doing anything, friend...your personal safety is most important, and who knows, it could have kicked off had you said something.
It's not nice to hear, and even though i've just ignored it over the years and still went to games, i can understand why it would put some...