I have had this problem and going close is a huge deer turn off.
They smell you real fast. If "nature calls" I would go farther away. But do some still hunting on your way there and on your way back. Try going somewhere that might push a deer through a natural funnel or up a field to your stand or blind.