should i polish enough off so the charge in the cap is on the nipple face?
Yes. The priming powder in the cap should be against the nipple. I generally cut a strip about 3/8" wide and 10" long from a sheet of 400 grit Emery paper to polish the nipple. Grip one end of the strip between thumb and forefinger of one hand, and the other end of the strip between the thumb and forefinger of the other hand, and polish the nipple the way you would polish the toe of your shoes with a rag.
SOMETHING ELSE TO CHECK: I assume we're talking about a sidelock here. You might want to pull the lock and check to make sure the internal screws are tight, and the moving parts are clean and
lightly oiled. Loose screws in there will cause the action to drag, reducing the force of the hammer strike.