What kind of fence?
Man, there should be a season limit on "hypothetical" threads.
If it is in PA, is the land posted? PA game animals are considered property of the state and not private landowners. Where I hunt, if land is not posted, it's considered open to hunt. That does not mean you won't get your ass kicked for going in there, but be assured that the police won't be involved in your punishment.
You know in your heart what is the right thing to do. Individual circumstances will dictate. Where I hunt, I would, but then again I know everybody, they know me, and are hunter friendly in the first place.