How many times did you have a chance to take him in the first year?
How many times did you have a chance to take him in the 2nd year?
How many times, last year?

I'm betting the number went down with each passing year (no pun intended). If it did (with all things equal....like how many times you hunted him, when you hunted him, etc.....)......then YES....he's becoming harder to kill. If it didn't go down.....I'd say you had an anomoly.

I hunt in a place with VERY little hunter pressure. I also keep trail cams out in the spring/summer to inventory the deer in my area. I see many deer on trail cam that I don't see when the season starts (with my eyes or anything else). The fact that I see less older deer makes it relatively easy for me to say they're harder to kill.