YES. I never recommend putting a quiver on the bow. Everytime you take an arrow out of the quiver it will need to be retuned!

Now if it is removable and you tune with it removed then put it on the bow just to transport, then take it off once at your stand, that is fine.
Your bow needs to balance level in your hand. When you hold the bow with the quiver on the bow, the bow probably leans to the left, then when you take it off, it probably doesn't lean as bad. This is causing the left and rights, most likely.