I have had this problem myself. The cottonwoods wouldn't be bad IF their trunks were vertical. It seems that the bigger they are, the more of an angle they grow. Either you feel like you are going to fall out of your stand or that you are laying backward against the tree. It sucks!
I opted for a smaller tree but like you said, it was almost too small and I swayed around a lot when the wind blew. One time it actually cost me a shot. Eventually the tree grew bigger but that takes a while.
You gotta do what you gotta do.
Possibly, select a smaller tree and don't go to high up. If you go this route, you'll have to sit extremely still because the deer will pick un your movement easier.
Good luck.