Do you have to match the color of the buckle to the color of the cantle on the sporran? For that matter, do you have to match the color of the belt to the color of the footwear ( no mixing brownshoes and black belt, or chrome cantle and gold buckle)? Obviously this depends in large part to the occasion and the formality of the event.

You probably get more attention for wearing a kilt than the average guy, so any "fashion discrepancy" is likely to be noticed.