"My question is...if I have been doing just fine without ever having a trigger job while hunting, is it worth having it done? I know it would help me shoot better and I would really like that, but how much do you guys feel it compromises safety, if any?"

In your case, since you are a hunter and are getting your game, I don't think you need to pay for a trigger job. A trigger job done by a competent gunsmith does not compromise safety at all, BUT there are some shooters out there who can't be trusted with a light trigger! It is not the gun, but the shooter I worry about!