From a British point of view,if you are not serving in the unit, then you don't wear the badge.Even a retired member of a unit would only,perhaps,wear his badge on Rememberance Day.