My plots don't show a lot of grazing either but the deer are there. My cameras take shots of them every day. I think the clover regenerates so quickly in the spring, unless you're out there all the time you won't notice the grazing.

I know when I mow this time of year it puts on about 6" of growth in a week or two.

Also, as mentioned earlier they don't have to go far to get young tender native plants. It keeps them from having to concrentrate on the clover as much as later in the summer and fall when the native forages get more mature and tough.