Another thought: what kind of clover? We use a blend of clovers, each has a period that the deer like it. If you have lots of corn and beans in the area that will pull your deer.

Without more info I would say keep the clover, overseed/frostseed it in the spring with a good blend of clover,12lb per acre. Add a forage brassica plot such as maxximum by either tilling part of your clover plot in late summer or adding another plot all together, that will help transition the deer from "green to green" when the clover sours after frost