I would think teeth wear would be a major problem if they ate shag bark hickory nuts. Those things are hard to crack with a hammer. Pig nut may be a possibility.

30 years of hunting and have not seen a deer eat a hickory nut.