Looking at photos from that era, a gentleman would have felt naked without some form of headgear so, as you say, a cap of some sort would be essential. Also the shirt collar would have been the stiff, detachable, variety and I don't know if regimental ties were worn in those days. Jacket styles tended to be a little more elaborate such as the Norfolk style and yours would probably have defined you in society then as a gillie or stalker with your choice of headgear being spot on. I seem to recall that sporans worn in those days were mostly the long, hairy variety or else some poor animal such as a badger, preserved for all time as a sporan. So again the Victorians would think that a plain leather sporan denoted an estate worker of some description. Of course the Victorians were extremely class-conscious and everyone was expected to know their place.