sounds like you have the same problem i once had when i got into scopes. make sure the rings and mount are tight. also what size rings do you have? i find the lower to the barrel the better. also when you shoot at 50yds. are you sighted in at 50yds? that might also be a factor. you also might want to try another ammo brand and grain size. my 30-06sprg is quite fond of the federal sierra gameking' s 165gr. 1.080" .
i find if i sight mine in at 50yds then shoot at 100yds they are not even close that i had at 50yds. also you might want to look into leupold mounts and rings. never had a problem with them since i bought them on either of my scoped guns.
they worked great even when i took a nasty dump on some icy roads logging trail last year. gun held it' s zero after the impact of the fall. also when power are you sighting in your rifle? 3X or higher? i don' t know if it matters but i sight mine in at the lowest setting.