RE: Am I Expecting Too Much?
If your rounds are from 3" to 6" from "hole to hole" that ain't bad! As long as it is consistent and you can "center" the pattern. Tighter would be nicer obviously, but being no more than 3" off of dead center of your aim at 100 yards will work on deer size game.
Now if you what you are describing is a multi-shot group (3 minimum in my opinion) opening to 6" from "center", that is not good.