I've actually seen guys use weights on the end so that it will drape over the shoulder with minimal movement. Weight will also need to be sewn-in on the back side to evenly disperse the "pull" over the shoulder. Put the brooch over the weights on the front, and it still looks like it's attached.

The best weight to use is the "lead tape" that guys use on golf clubs. Many sporting goods and department stores stock this stuff in the Golf area. You should be able to stitch-in a couple "tack" spots to hold it firmly.

This would need to be inside and out of the wind. If you have the pocket, the weights will help there too.