The White Rifle system is a slip fit conical system. They are supposed to load easy. I believe if you held the rifle pointed at the ground and shook it enough you could shake it off the load. Why don't you try by loading it, place a piece of pine board down, and then holding the muzzle end down, LIGHTLY tap the muzzle on the board and see if they come off the charge.
Also something I do when hunting. Is every so often I check the load. No matter what kind of bullet I carry. It takes but a few seconds for a lot of piece of mind to take the ramrod and check the load by making sure it is on the charge.
I use 65 grains of Triple Seven 2f powder and that conical and get excellent accuracy. I have gone as high as 80 grains and still got excellent accuracy. But for deer, 65 grains is plenty of power.