Without chrono data and the powder variables, im really not sure why that would happen. I consider powder as part of the ejecta weight. Since most of the powder doesn't usually leave the barrel, IMO recoil starts at ignition and probably ends in under 2 milisec.
The only test ive tried on POI vs powder charge was with BH209. 100gr vs 110gr at 100yards. The 110gr load was consistently 3/4-1" higher in my 45 with a 200gr SST. That was on a sled too so it not apples to apples.