Differing clan badge from other side of the families Tartan

[ctbuchanan]

Not in my opinion. I never wear my clan badge on my kilt and I often wear an Irish pin as a nod to the Irish side of the family.
And frankly I'm not sure anyone would notice.
Although my brother wears a turtle pin on his kilt (it is his spirit animal) and a indigenous Canadian once asked him if he had Mohawk blood. Apparently they are often call the Turtle Clan. They both had a good laugh over it.

If anyone asks you, just tell them why you are wearing it.