I think that in terms of legal technicality and general philosophy the game is the property of the state. Someone who thinks that the deer is "mine," should see what happens if they decide to shoot "their" deer in mid-august without a permit.

If someone tresspassers on you and kills a deer, they committing two separate crimes. One against you as the property owner by crossing on to your land, and the other against the general public by unlawfully attempting to take a public resource in an illegal manner.