How odd that people should ask that question when you're wearing an overcoat and a hat. As you say, you've covered your upper body well with a warm fabric. It's said that 40% of our body heat is lost with an uncovered head.

On the other hand, I did stop a young guy last week, who was out in 20+ degree weather with shorts and flip flops (no hat) to ask "the question." Was I wrong?