It depends on which state you live in as to who legally "Owns" the deer. In Texas the deer legally belong to the people of the state, as opposed to the state itself. This protects the state from liability when cars are destroyed from people hitting them.
The only way a you can own a deer in TX is if you have a breeders permit and you may then only own the deer you have registered as your breeders. If a random deer walks onto your place or gets into your breeder pen, you do not own him.
Most folks say "My deer" because most ranchers and farmers put out feed for them or take care of the habitat that the deer live in.
I've been guilty of saying "my deer" too. However, they are not truly mine. I can sell you the right to hunt on my place but the deer do not actually belong to me.