That's the great thing about hunting--you can do it all. Personally, I couldn't give up deer hunting and am amazed that you even considered it. I couldn't give it up myself.
Perhaps over time, you can add another rifle or 2 to increase your hunting seasons and game hunted. It's pretty rare for hunters not to acquire different guns and hunt additional species. The smaller critters are fun to hunt also and usually have longer seasons. Plus they make good hunting practice and get you out in the field all year long.