Going from 4" vanes to 4" feathers will drop your arrow weight about 18 grains or thereabouts. You'll get a bit faster arrow flight out of the bow and a good long way downrange, well beyond 20 yards. Drag is not a problem if you don't wrap a hellish amount of helical on the feathers. Most people I know use far too much helical.

If your broadheads are a little erratic with 4", then try 5". Trial and error is the only real way to tell. Generally though, the larger diameter and the more blades (more blade surface area) a broadhead has, the more fletch it's going to need to control it.