I had the same problem, I weigh 145 lbs.,my choice was made for me when the only size available was a L/EXL. I found that I could adjust the fit enough so that it worked wearing light clothing for a MD bowhunt, lately hunting in MA with heavier clothing works too.
One of my friends has the smaller size and after looking at it I was glad that I had the bigger one.
Her vest came with an oval shape clip to attach the strap to the tree, the larger size does not, you have to thread it thru the strap. While muzzleloader hunting in MD. my hat blew off and after I retrieved it and was rethreading the strap, it cost me a nice buck, but that's a whole nother story!
So when I returned home I went to a mt. climbing store and got a heavy duty locking carabiner to use to avoid the problem. Probably not recommended for heavier hunters but as long asthe carabiner is big enough to use correctly I don't see a problem.
Hope this helps.