I don't know about Victorian buttons, but even in Japanese culture the men cross clothing left over right (kimono, gi, etc.), and women right over left, just as in Western culture. I don't know why, it's just the way it is.

Andrew.