How much will the diameter of your peep sight effect accuracy?

Last year I had a really small peep. During the late hours of the day, I had a hard time seeing my pins. After the season I put a large diameter peep on. I just have started shooting again, but my groups are much worse than last year. At first, I attributed it taking a few months off, but now I still can't figure out the problem and changing the peep is the only adjustment I've made in since last year.

Thoughts