Echoing Arthur P; forget the speed. These days, speed is so highly overrated other more important aspects have taken a back seat. You gotta use what you gotta have.

Rather than being concerned about loss of speed, I would be watchful that the spine of your shaft does not become too stiff once weight of any type is added that skims off energy. Changing of shaft spine is something many shooters hardly consider when adding weight to the string.