As already mentioned...a warm hat and keeping the upper body warm (jacket, jumper) and wicking t-shirt beneath is important.

I don't think anyone else mentioned gloves. Keeping the hands warm is also essential.

I was out for a late-day hike this past week in a 5-yard wool "beater". Wicking thermal long-sleeved t-shirt beneath a sweater, wool toboggan hat, and wool gloves. By twilight temp had dropped to near freezing and I was heading home. Ran into a search party that was looking for a family who had taken a wrong turn. Back into the woods I went for another two hours. I was not at all uncomfortable, much to the amazement of the SAR guys. (of course, constant hiking keeps the legs warm as well )