I have taken 3 deer with .50 PowerBelts. Closest was 12 yards, furthest was 65. All were hit just behind the shoulder, halfway up the body. All dropped like they were struck by lightning. Well, one did crawl about 10 feet. All had impressive exit holes. My experience, small as it is, tells me that you are pushing a too light bullet too fast.
I used, and still have some left, the 405 grainer in front of 70 grains of either Goex 2F, or Pyrodex RS. This duplicates, to all intents and purposes, the original 45-70 rifle load. I don't think anyone can tell me that this load isn't going to work on a whitetail. Heck, it easily drops Bison.
So, I think you should either go to a heavier bullet, or drop the powder charge.
Just my opinion.