I'd start with a hunter ed course. You can learn many basics, and it all stems from the basics...
If you've any friends with guns, you might ask if they'd take you somewhere to get some shooting in. Offer to pay for ammunition and fuel, and see about joining in on hunts if possible, if only to tag along and watch.
You obviously need a gun to get started, and since you're just starting out, a 12 (or 20) gauge pump shotgun can get you far. From doves to geese, squirrels to deer, you can do a lot with it. I'd look into the Remington 870, Mossberg 500, and Benelli Nova. By look into, I mean that they all are solid performers or I wouldn't recommend them, but feel 'em. See how they fit you, and decide from there.
When you finally have your gun, learn about it. Read your manual, learn how to disassemble it, learn how to use chokes to control shot pattern, and spend some time shooting targets.
Apparel is sort of important, but don't think you need to get $1,000 sets of camo. Jeans and a sweatshirt will get a guy far for small game, though I'd grab a vest with a game bag. Obviously, dress for the weather.
Ammunition must be matched to the game. In general, use smaller shot size for smaller game, and vice versa. You'll learn about shot size in hunter ed. You'll probably want to use #6-8 shot on squirrels and rabbits... I carry ammunition on a belt.
Be aware of how to field dress the animals you're going to kill.
After some simple preparation, you're dressed for the day and have your gun and ammunition, the ability to hit what you're shooting at, a knife or two, etc.... hit the outdoors and try to get some stuff shot. You can learn about the good spots in your area and how to approach situations with some thought and you'll only become better.
In the future, you may want to add a 22 rimfire rifle to your collection. From there, perhaps a centerfire rifle, maybe a muzzleloader, a bow, whatever. Some guns you'll want more than one of, just because. Some guys stick with small game on occasion, though a lot of us hunt about anything as often as possible.