As long as a sweater is appropriate, then don't be afraid of wearing one. They can look quite smart with the kilt.