I'd go with the .45 ACP if you're looking for a defense pistol. I love my 9mm (surplus Walther P-1), but it just doesn't have the punch I want in a self-defense sidearm.

On the other hand, 9mm's are easy to shoot and ammo is cheap and plentiful. So if you're looking for a plinking/target pistol, that's your choice.

My rule of thumb is it's better to have more than less, because you never know when some thug is going to try storming your house. Now that I've already gotten a 9mm to "play" with, I wouln't go any smaller than the .40 S&W.