ORIGINAL: SwampCollie
Thats exactly what you'll need to do. Most bow shops will have a coffee can full of spacers and screws of various sizes. In the past, I have even had to manufacture spacers out of (brilliant really) cut down pieces of carbon arrow shafts. Trick is not breaking the shaft..... but there are plenty of spacers made specifically for what you trying to accomplish.
I used a piece of BLACK small bore 1/4" PVC pipe to set up one of my bows this same way, Not sure I'd use arrow shaft though, you could probably run to your local hardware store and get a great assortment of things to do the job, nice little rubber bushings would probably be great, but when I had this stuff laying around, it was perfect for my needs.