I've been doing this a long time and have my thoughts on it.
In most cases the rifle is more accurate than the shooter is. It takes a long time and a lot of practice before a shooter can shoot better than most rifles do.
Sometimes it is fit, if it doesn't fit right you are unlikely to repeat exactly and gain consistency.
Parallax when using a scope.
Picking up bad habits as you shoot. The key is consistency, right down to how you breath and even your heart rate. A little cant can cause big problems. How tight you hold the rifle to your shoulder can also have an effect. A whole bunch of bad habits can creep in.
My biggest gains in accuracy are (for me) from the trigger. Using a set trigger really shrank my groups.
Optics, using better optics and finding the right reticle also had a large effect on my groups.
When I finally learned about eye relief on a scope and learned about parallax. They advertise parallax free optics or parallax adjustable optics. To be honest, the advertisements should be parallax reducing.
And eventually finding which ammo that particular rifle preferred. Before you could even start to determine this, you had to be a better shot than the rifle, which took a great deal of time and practice.
I've had two rifles I just shot really well, I really don't think it was the rifles so much as they just fit well and we worked well together. One is my SSG 69, which you would kind of expect. But even the 69 will change point of impact if I let it get hot. The other was a Winchester model 70 extra lite. You'd think that skinny barrel would mess with the groups, it may in theory, but in practice it didn't seem to matter at all.
The worst shooter I ever had was a mini 14, I never did shoot any better than 3-4 inch groups at a hundred yards with it.
I still have an old Stevens in .222, it is pretty much used up, the lands and grooves are almost gone. It still shoots one inch groups at a hundred yards. I'm consistent with it and even with all it's flaws, it still shoots consistently.