I find it amusing that he waited until you were no longer at the table to make a disparaging remark about you.

Obviously he was concerned about your reaction might be if you heard him say it, so it sounds like he is in fear of you. (Must be the kilt.)

Keep his feet to the fire, especially since he insulted your wife.