"Is it worth it for me to buy all of the stuff to reload rifle bullets if I am getting the performance that I want out of factory loads? "

How much rifle shooting do you do? If your rifle shooting consists of merely zeroing in then shooting a round or five at game every fall, then use factory ammo. If you shoot a lot, and/or want the ability to develop special purpose or tailored ammo, then you need to be able to load your own....