Since it's yours (you served in the unit), I would say why not?

I know of a piper that wears his U.S. Army officer's cap badge on his Glengarry. With nearly everyone wearing a badge of some kind on their headgear, it's not extremely obvious that it's not a clansman's badge (strap-and-buckle) or armiger's circlet, but it is a conversation starter for those that know what it is.