I know what you mean, I shoot a Hoyt Magnatech and get lots of grief that it's not a top of the line bow and it has a cast rather than machined riser. Well it shoots just fine for me and I don't see enough of a difference to even be really tempted by a new bow (and this coming from an admitted gear junkie)!!

Find a bow you like and learn to shoot it well, that's all that really matters.