.25-06 would fit the bill. Shoots very flat and has plent of punch with little recoil.

But for $500 I think you may have a hard time putting a rig together, especially if you are planning on buying new. Plan on spending around $200 for a scope if you are planning on using a scoped rifle. Add in maybe $50 for the rings, and your down to $250 on the rifle. If you keep your eyes open at pawn shops and go to a gun show or two, you might find a good deal on a gun or scope. And remember, you can buy a combo only for the gun and the scope, keep one and sell the other. Sometimes you get lucky and the pawn shop doesn't know much about the scope.