I guess I'm kind of a wuss too, although I've been bumped from my intended spot too many times to count, I still feel guilty if I do it to others. I'd try it one morning to see what kind of a reaction you'll get. MOST hunters I've ever come across on public land almost start to think that 'their spot' is only 'theirs', and will cuss you out if you dare come near it, regardless who gets their first.
Give it a try and be prepared for a pellet rainshower!
Chad