I'd stick with Pyrodex R/S, or use BP. I'd try dropping back to 50 gr and see what happens, then bump it up to 80 gr and see if it better or worse.
For PRB (50 cal) I usually start off at 50 gr then go up in 15 gr (50 - 65 - 80 - 95)increments to see if it likes mild or hot loads. Once you find the general range of best performance, then you can "fine tune" with 5 grain changes.
If you think the bore might be rough it wouldn't hurt to polish it lightly, but I wouldn't get carried away. A change in patch thickness or ball diameter can sometimes make a big change, too.
Bottom line - buy lots of powder, different patches, different dia ball, and get out and shoot a lot! It's fun, not work!!!!!!!!!!