Bomber jacket, or skip the jacket altogether and just wear the turtleneck sweater. I don't have any opposition toward your "views on casual use of traditional garments", I'm not a very formal guy myself, but I always thought that wearing a sport coat with an open collar looks a bit pretentious.

Jamie