RE: clover vs alfalfa
I have not heard of anyone doing a side by side scientific deer preference comparison of clover and alfalfa so I don' t want to make a comment on that. here may be a few other thoughts.
1) The alfalfa field probably has been around for a few seasons and the deer are use to it as preferred food source. They may not have found your plot yet.
2) Check the surrounding habitat around your plot and the alfalfa field. The neighbors field may be in a better location then your plot.
3) Fertilization! The more fertilizer put down (up to a point) the more lush growth will be present and the greater deer preference. So who did a better job of fertilizing, you or your neighbor?
4) The alfalfa is taller then your plot. This will lead to more deer use especially by doe/fawns earlier in the year.
5) A combination of one or more of above.
Don' t give up on your plot yet. With less deer use now, the clover will have a chance to become establish. Keep it mowed and fertilized and deer should come. In a few years and if there is an abundance of alfalfa in your area, you may want to convert your field into a late year plot by planting corn, milo or something similar, that the deer will have accessed to when the alfalfa has frozen out. Good luck.
greg