I prefer the buckle somewhere on the back. It should fall in the small of the back which would prevent it from being uncomfortable. If you really feel the buckle you may be wearing it too high or too low.

The buckle on the side could create more friction on the kilt or look off balance. I have seen some wear sporran belts (no snap hooks) with the buckle right down by the sporran and it looked better than having the buckle up on the hip.