I think the velcro may exacerbate it, but the issue behind it is the design of the buckle itself. The side that has the bar usually stands further out than the side with the hook. Even laying it flat on a table will illustrate the problem. I have three kilt buckles, all from different places, and they all do it.