Cal -
Yep, I would attach it and in some manner make it removable (maybe snap clips or something durable but quiet) so you can take it off when you get to the tree.
You may already know about this - but have you seen the 'paracord turkey tote'? If not, it's a simple woven paracord rope with slip knots, that some hunters are raving about for carrying out birds. I usually slip the gobbler in my vest too, but my wife is making a tote for me to use this spring - it does have the advantage of hanging the bird off a tree when you stop to take a break (even in the vest, those 20+lb birds aren't getting any lighter as I get older).