I'd still say you're doing fine. The typical group size at 50 yards is around 1/2" - 1" with a scope and a rock solid rest, so assuming that you're getting 1.5-2" groups, your aiming error is still less than an inch. Good enough for deer.

Mike