I would think it would mess up your zero, too. I would never do it to begin with because of the potential for damage, and it would seem to me require a new sight-in session. It might be the difference between a clean kill or a miss--or, worse, a wounded deer.
I'd also be leery of that thing falling off and banging up the rifle as well as what it might do to one's chances of being able to make a quick follow-up shot. It looks clumsy.