If it matters I say, don't let what others think and say force you to do anything you must be in control of yourself or you will find yourself pushed arround all your life. If you choose to leave your employer that is your choice but let the HR no why in a formal letter of resignation so that hopefully you can save someone else from having to experience what you have.

Just my 2 cents take it or leave it your choice.