Yodhan,
I grew up in H'Burg and now live 1/2 hour away. The town has a very diverse population. Recently in the last 10 years or so folks have moved into the area from Asia, Middle East, Eastern Europe and South of the border-to give you an example the Rockingham County School system accounts for over 30+ different languages being spoken by the students.
You also have James Madison Uni, Eastern Mennonite Uni and Bridgewater College within 10 miles of each other.

The Burg is still "blue collar" but the folks are really pretty nice.
I'm kilted all the time in Harrisonburg and have never had any trouble-just some strange looks from the macho hispanic guys, but no big deal (I actually think they admire a man with enough back-bone to wear a kilt).

You should have no trouble.