You don't say what powder charges you've tried. The bullet is only half of the equation. You can work up a charge for a given bullet, or you can find a bullet that will work with a given charge.

I'm not a fan of pellets. Just don't buy into the notion that there is any practical advantage, and they cost more. They also make it more difficult to vary your powder charge incrementally until you determine the optimal charge for a certain bullet. I shoot lead conicals that I make, so in working up a load, I only vary the charge. I suspect that if you start at a "light" charge and work up in increments of 5 grains, you'll see a trend of good accuracy up to a point and then group size will start to increase. When accuracy is no longer "acceptable" you know you've reached the limit for that projectile.

You also don't say what your groups have been. They may be ok, or they may be terrible. Point is, you may be expecting too much. You'd be right to be aggravated at 6" groups at 100 yds, but 3" groups are another story. That may be the best you'll do with a certain gun/load.