If someone shot at me, hell yes I would defend myself. I carry a CCW with me during bow season for this very reason.
Most states it's illegal to carry a firearm while hunting. I dout just because you have a CCW permit it would allow you. I hunted with a county swat teamleader for yearswhom was my next door neighbor and he would leave his weapon in the vehicle. Now this was 25 years ago and things might have changed but I would check into it if you haven't already.