My first thought was that the sights on the .22 must be better suited to your eyes than the sights on the .54.

I know as I get older those 100 yard groups sure opened up for me. I used to shoot about 1 MOA groups at 100 yards, and today a 4" group keeps me smiling. (with open sights - with a good scope groups remain tight)