The answer is - you need to dedicate a gun - just for hunting deer.
Go out and buy a couple of different types of slugs, different manufacturers and take it to the shooting range and find out which slugs your shotgun shoots best.
I don't know if I said it here or not, but often times a smooth bore shotgun barrel works just as well as a rifled shotgun barrel.
When you are throwing a big chunk of lead like that - more then 50 yards, it is hard to predict from one time to the next where it is going to hit - due to factors beyond your control such as wind speed, outside temperatures, shooting up hill vs down hill etc.
Most times it helps to have a shotgun where the barrel is physically pinned to the action. That helps to keep the barrel from moving around.
Then when you do find the type of slug that shoots best in your shotgun, you need to go back to the store where you bought those slugs and buy 10 or more boxes of those slugs. The same slug, the same lot number - everything identical to the ones that you used to sight in your shotgun. Keep them on hand and only use them for deer. If you shoot one box a year, you have a 10 year supply.
The initial expense is pretty huge, but the rewards down the road is that you are not spending $20 a year to sight in the same shotgun every year. If you have a really good scope and a really good rail type system to hold the scope and if you keep your firearm in good condition and not abuse it - letting it ride on the floor of the truck, dropping it on the ground, falling while walking, the one sight in might last the entire 10 years.
Since you probably live in a area of the country that does not allow center-fire rifles for hunting deer or hogs, this is probably your best bet.