Use of neighboring land is obvious = trespassing. But a lot of trespassing laws require posting or warning before prosecution, so check on that and do what you need to do first. You should also be able to recover for actual damage done to your land. Some states have laws giving double or triple damage for things like willful timber damage.

As for use of his land, there are only about 2 approaches: (1) noise ordinance: I have seen that work where a dirt track was put in next to a horse farm (for racing), and it was ruining the horses, (2) environmental issues: If he is digging ditches, filling in streams, etc. then he may be violating certain Corps. of Engineers wetlands law (like 404 permits you need to build a neighborhood on land classified as wetlands). It doesn't take too much for land to meet wetland, but you will at least need some kind of water moving through there.

Filing suit could get expensive. So at a minimum I would pursue the trespassing thing first. If he violates it you could get an injunction, and then have the trespassers arrested.

Good luck.