I hope you recover you deer, without too much hassle.
What I would have done is to ask the guy (last time you were with him) for permission to retreive ANY deer that you shot that crossed onto his property - so you could avoid any potential hassle with his wife. Sounds like you didn't, but he probably would have said it was ok. Good luck, I hope to see some pics soon.