I had the same situation. My buddy's property backs up to Valley Forge National Park. Tons of deer. He has a wooded lot of a couple acres. The neighbors on one side are kind of animal lovers. It's a tough one. You have a small sliver of opportunity and you need to make that shot count.
Just make sure you shoot the deer on his property and get permission to rertrieve it if it runs off the property.
I'd be choosey, too. Not worth the hassle for a little buck.
You could very possibly shoot a monster. But it's a trade-off. Not quite as much the hunting experience as a big farm or the forest.