Less traditional than the two above suggestions, but for very casual wear a 1/4 zip fleece is quite handy.

They're light weight, inexpensive (generally speaking), and seem to have a very wide range of temperatures that they are comfortable in. As a bonus, many of them seem to be cut shorter than most jackets, so they don't spill down over your sporran.

As an added bonus, you might get mistaken for Duncan MacKay (of Monarch of the Glen)

ith: