Set up a paper target with a bulls eye on it at the point where the missed deer stood and shoot 5 rounds at the bulls eye. If the 5 shots hit within about 1" to 1.5" (corresponding to 2" to 3" diameter circle) of the center of the bulls eye, you just muffed the shot -- "buck fever." If each of the 5 shots are NOT so disposed, either you are not a very skilled marksman or your weapon is not accurate.

If you are the culpret, don't get depressed. Shooting is a skill which is developed and improved through practice, not a genetic trait inherited at birth. Spend time at the shooting range working at shooting accurately and you will get plenty good enough to take deer reliably at 100 yards.

If your weapon is the culpret, several problems are possible. You scope may move in the scope rings. Think of shooting your rifle as similar to delivering a sledge hammer blow to your rifle -- this might cause your scope to slip in its rings if the screws aren't down tight or other problems exist. I have experienced this problem before. Your reticle could be broken loose from one or more mechanical mounting points within your scope. Either of these problems can lead to the rifle not maintaining a constant targeting point. Be aware that if you change ammunition -- brand, bullet grain weight, bullet style, etc. -- this will very likely change the targeting point of your rifle. Sight-in your rifle with the ammunition you will hunt with and don't change. If your rifle barrel is dirty -- old powder residue, copper build up from shooting copper jacketed bullets through the barrel -- this can cause inaccuracy. If the barrel is corroded from long use or failure to clean, the rifle may be inaccurate. There may be other issues I'm not identifying.

But the main issue is to run the test with a steady target at that distance with your current rig and see what happens.