The only time I've ever had an issue was with someone who was against the blackwatch being worn (as the regiment in question had been used to put down several of the Irish uprisings over the centuries). So, out of that sentiment, I tend to avoid it, but I have worn others with no issue.

Certainly less silly than the shamrock spring headgear things I have seen today.