It would definitely cross my mind. It all depends on what you have the deer for I guess. If it's to eventually have hunts there, then, why put up the fence. If it's for your own enjoyment of having the deer there to see your hard work in the end, then fence it. If you raise them so that only you are able to shoot the deer that you raise, I would fence it as well. That's my opinion though.