Did you calibrate you GPS?
What is the model ?
Comparing the reading from a topo map with the cordinates marked on the GPS your location will usually be a ' meter or two' different...but not a whole lot, as you've mentioned.

Try calibrating your GPS, using Magnetic North and see what happens.