Yes, there is a difference. More difference between your $100 pair and the $300-400 bino's than between the $300-400's and the high dollar stuff, IMO.
Even going down to the $200 range you can find some very servicable bino's. Nilkon is good, Leopold is good, I have a pair of Minox 8x42 that I paid $150 for that are as good as the 2 pairs of Leopold bino's I have.
Try looking thru the differnt posts here about what bino to get, lots of good sugestions and links to vendors.