OK, as my computer wouldn't let me post last night at home...
A lot of ladies handbags use pretty strong magnetic clasps without damaging the cards, (just their husbands wallet), so I wouldn't have thought it'd do too much damage.