Is there really that much of a difference in the 40 & 45?
Yes and no. Its a double edged sword if you ask me.
how big is that difference really...
The wise-arse answer is 5/100ths of an inch, but thats likely not what you are looking for. The real difference is in velocity. This is where you have to subscribe to a school of thought...do I want smaller/faster or larger/slower? Just like deciding on a rifle bullet caliber or arrow weight. The faster one will be flatter shooting, have more bark (and in this case more recoil too), or do I want something with a little heavier bullet, larger wound channel, but slower and not as flat shooting?
is the difference worth it?
That depends entirely on which direction you are headed and what school of thought you are now a member of. I study at the school of 230. (ie: .45). I prefer a little bigger, slower bullet. Never had to try it out on two legged targets, and Lord save me I hope I never do either. I just like having a little extra diameter to make up for my personal inaccuracies with a handgun...as if 5/100" will do that anyway, but it helps me sleep at night.