Hah! I SO know what you're talking about! For a long time, I did that every ten seconds. It lessened over time and I got into the habit of noticing I was doing it. Halfway through the motion, I'd change it to a brushing off of imaginary specs or tugging the edges of the front apron.
Now I hook my thumbs in my belt and strike a pose.
I find a beer in one hand and a smoke in the other obviates the problem.