Do it in the middle of the day if you can when they don't move as much and I think you will be ok. It is much better than the alternitive. Hunting with a gun that you have no idea where the bullet is going to go is just reckless.
I totally agree with Paul... Mid-day is the best time, and I don't think you'll have much to worry about. Deer don't have very long memories, and a few gunshots won't drive them into the next county anyway.
I commend you for asking the question, and caring about your bullets hitting their mark. You can get your gun boresighted at any gunshop, or even buy a fairly inexpensive one at Waly-World, and that will at least "put you on the paper", and cut down on the amount of shooting.
Good luck on your upcoming hunt!