I think the most casual option by far was included in lethearen's list, and I'd like to give it another vote - the flat cap.

I'm not a fan of flatcaps worn with the kilt, but if you're looking for a casual tartan accessory to wear when NOT kilted, I don't think you can beat its versatility.