Here's a suggestion...

Stop worrying about whether it's 'right' or 'wrong' by some set of arbitrary rules, and decide if what you're wanting to do will be aesthetically pleasing. All of The Rules about wearing tartan this or that are descended directly from whether the combination is aesthetically pleasing.

If it's gonna clash, don't do it. If you'll look like a tryhard, don't do it. If it'll look good, just make sure it's not the tartans of the MacDonalds and the MacLeods of Skye (for example), and...as much as people around here say anything goes, there's a limit...so if you can, try to have a reason for what you're doing, if it's out of the ordinary.