Nathan, I have a black velvet Balmoral as well, in the event I feel a bit overdressed I just unbutton the top two buttons as this seems to lessen the formality of the Balmoral but I doubt I'd worry. I first noticed John Macleod of Macleod, the late Chief, doing this when he wore his and it seemed to work well to "soften" the look. You can get a look at a photo of him here.
http://www.clan-macleod-scotland.org.uk/tartans