Honestly, the .22 is the best caliber to learn to shoot with.
What you're experiencing is how a gun fits you. You need to find some rifles and hold them. When you find one that fits you, you will know it. The right gun for you makes a big difference in how you shoot.
Unfortunately, I'm not as good as others here when it comes to describing what you want to look for. I just know from experience that when I hold a gun that fits me, it's a natural extension of myself. It moves with me naturally making lining up the rifle to make a shot with ease.