Not that hard, or it wasn't for me.

I expected to shake horribly when I shot my first deer, having always shot them 100+ yards with the rifle.

Your instincts kick in when its time to get down to business though. You'll shake a bit before the shot, but once you draw back you'll be golden. Rock solid until you release that arrow then the adrenaline really flows. You'll have to sit down after the shot or risk falling out of the tree.