Not sure if this still holds true with today's bows but the rule of thumb used to be that you would lose 1 fps for each 5 gr of arrow weight increase. So you're going up 50 gr would mean you would shoot about 10 fps slower. Not much difference in my opinion. If your current set up is shooting fine for you that's great. But a plus to a heavier arrow is more KE and usually a quieter bow.
Don't get hyped up on the speed thing. Us old guys killed millions of deer using longbows and recurves the rarely exceeded 180 fps. Just put that broadhead where it needs to be and you will have your animal.