Go to the local barbar shop and ask them to save you a trash bag full of human hair. One day when you know he's not there, go spread the hair around the bait pile, not too much though or he'll see it. I've heard that deer will stay away from human hair clippings. I'd have to agree since they probably smell like shampoo.
Don't get shot doing it!