Are you using a peep? Do you shoot with both eyes open?

The reason I ask is, if you shoot with both eyes open (as I always did when I used a peep, don't use one anymore) and you have a cross dominant eye (i.e. left eye dominant shooting right handed), it is possible to let your left eye take over, which would cause you to shoot way left. Perhaps this is not happening at 20, but is showing up as you get farther back?

Anyway, just my thoughts. If you shoot with your off eye shut, never mind ...