You don't need a large powder charge to kill deer...I use 80grs FFF Goex in both my Knight and my .54 flintlock...Remember the 45/70 used 70grs of FF powder and a 405gr bullet...
I would change bullets though...A poly tipped bullet opens up better with more velocity...I'd go to a Hornady XTP .430 240gr hollow point...Put 60grs 777 behind it and it will do all that needs to be done within 100 yards...