I say take him, for the time to spend with grandpa and because he will be a man one day. There is going to be a time in his life when his eyes will be opened to what men are and how they are, I find it best to at least introduce him in an environment you know, enjoy and trust. Plus, your father may end up taking him with him to hunt and if need be help him get the deer out with his age and all.
You may make a call to your buds and father though just to give them a heads up, instead of plopping it on them, a FYI, we are welcoming him to the man club.