I've used both hunting deer with a .44 Mag Redhawk. Never quite got used to the odd balance (and didn't have time to get used to it at the range first) with the scope. I found that I just couldn't react quickly enough at handgun range and sold the scope. I did connect with a doe a year later with iron sights.

I've also used a couple different "red dots", ranging from the garden variety Tascos on handguns to EOTechs on service rifles. Target shooting with a red dot at handgun range was lots of fun, but I think I'd have had the same issue in the field as I did with the scoped Redhawk. While an EOTech is great in bad Ahmed's house, it's not so good when he heard you coming and is running away with his AKacross the watermelon field 200 meters away UNLESS you have the magnifier (which I didn't). That "little" dot covers a lot of target as the range increases.

All I can say is try it and see for yourself. You may be more comfortable than I was hunting deer (etc.) with a scope. I do use a scoped XP-100 in .223 for prairie dogs on occasion, but that's a bit different environment than deer.