Absolutely agree with you here.
I have terribly thin hair and, though the cold doesn't bother me at all, I wear a beanie hat outdoors in winter to prevent headaches. And, in summer with heat which really does bother me, I wear a cowboy hat to keep me cool and shaded. (Even though I don't look well in it.)

Since I was a teenager I've had friends make fun of me for doffing my hat and coat indoors when they always kept their baseball caps on and for taking my hat off to speak to anyone outdoors that I'm not terribly familiar with.
It's just, to put it simply, polite. And I think there needs to be more of that these days.

Good on you.