70-80 grs of Goex FFF is plenty for deer...I used to use 75 in my .45 and use 80 in my .54...At one time I was using 120, then 100grs in my .54...I soon found out that patch burnout was a problem and I went through the stage of using wads, hornets nest, etc between the powder and ball...After killing a few deer with 80grs, I found out that's all I needed...My gun also preforms best with a clean barrel...You might also try experimenting with different lubes and if it is hard to load the 2nd shot with ..018 patching, try the 2nd with a dirty barrel and a .015 patch, the powder residue in the barrel helps to act as a buffer between the patch and powder...

Your 80gr loading is just fine for deer, you might drop down to 75gr and see if the group tightens up a bit...