Add some bleats mixed in with your grunts. A rutting buck doesn't necessarily come to a grunt call all the time, obviously but if you mix it up with a little sweet doe music it might just perk his interest enough to come check it out. Here, check this call site out:

http://www.bowsite.com/bowsite/features/practical_bowhunter/deergrunting/deergrunt.html

Are you sure they were bucks grunting back or other hunters?

Rarely have I ever had the opportunity to have a calling match with a whitetail. It's not like calling turkeys. Perhaps they were grunting and chasing which had nothing to do with your calling. Either way, mix in some bleats and contact calls.