RE: my imagination?
Chiefheadhunter is probably right on but 1 more thing is lighting.Is the lighting the same.Are you shooting into the sun at 1 place and have the sun to your back at another.If so,then you might be aligning slightly different because you have more light.This is one of the reasons people really like the round housings on their sights.They can move them and adjust the size of the peep so they get a good bit of light around the housing.
I used to have the same problem as you and it was a combination of not having my elbow straight behind me on my release hand and not having the sight extended far enough to get enough light to center my scope housing.