I wear a lounge suit to church, as I always have done, to church (Anglican) in the West of Ireland. I can remember being admonished for not wearing a tie one week. I now realise I am one of two who dresses like this and everyone else wears sweaters, slacks, track suits, etc.
I'm not a big Margaret Thatcher person, but I do empathise her father's wisdom that one should not do something because that is what everyone else does.
John