Most likely because your kilt has a tannish hue, making your outfit look a bit like the infamous "Black and Tans" of the Irish War of Independence.

As Wikipedia puts it, "The term Black and Tans refers to the Royal Irish Constabulary Reserve Force, which was one of two paramilitary forces employed by the Royal Irish Constabulary from 1920 to 1921, to suppress revolution in Ireland. Although it was established to target the Irish Republican Army, it became notorious through its numerous attacks on the Irish civilian population. However, Black and Tans very often refers to both the RIC Reserve Force and the other RIC force raised for the Irish War of Independence, the RIC Auxiliary Division."

The Black and Tans were so named because of their less-than-uniform uniforms, consisting at first of khaki British Army pants and dark blue or dark green police shirts. When shortages of British Army uniform shirts and caps cleared up, the Black and Tans wore regular khaki uniforms, but the name stuck.

So in future, you could perhaps respond along the lines of, "No, any true Irishman would know that the Black and Tans didn't really wear black."