Yes, you can grow anything if you are lucky and your soil has the physical properties necessary to germinate and support the plant you want to germinate and grow. However, if you are not lucky and your soil does not have the those properties required for growth, then you will have spent a lot of time, money and probably frustration when the plot either fails or struggles to grow.
Let me give you a real example of why it is important to know what your soil can support. Here in southern Michigan, our soils are generally pretty good for most crops. Typically our agricultural soils are relatively fertile, drain well, have levels of nitrogen, phosphate, and potassium that are relatively easy to regulate with moderate applications of fertilizers. I live near a nature center that recently received a donation to develop an organic community garden with about 50 individual garden plots (20'x20') for members to rent for vegetables. Being an avid gardener for the better part of the last 30 years and an involved member of the nature center, I was interested in a plot the first year it was opening and asked if a soil analysis had been taken. It had not and no one was even aware that soil conditions were even a factor since farming is common in our county. So, I decided to sample it myself and sent the soil to our agricultural extension service. It showed the rates of nitrogen phosphate, and potassium I needed to put on my plot in the numer of pounder per square foot and it wasn't too bad. However, one part of the soil analysis popped out as a red flag. The soil's pH is 5.8 (7 is neutral). Most of our agricultural soils are around 6.5-6.7 so a 5.8 means we have "acidic" soil conditions.
What do these acidic soil conditions mean? Every plant has a range of pH it needs for germination and promote healthy plant growth. Too high or too low and the seed may not germinate and if it does germinate the plant will struggle and may not survive. Having received the information and presented the importance that pH plays in healthy plants to the nature center garden committee, the they more or less ignored it since they had received the money, had already picked out the site and layed it all out. But the crazy thing was that they left those 50 garden plot renters in the dark and didn't tell everyone. Many of the seeds sat in the ground and wouldn't germinate and if they did germinate they only grew a few inches tall. Some plants that tolerate more acidic soils did ok. A lot of transplanted vegetables struggled...some died, others grew a fraction of their normal size. And the unsuspecting gardeners were frustrated thinking that they just were a failure at this whole gardening thing (many were urban people who had never gardened before).
There are ways to alter pH just as there are ways to alter other components of the soil so that it will support whatever it is a person wants to grow. A 20'x20' plot is relatively easy to manage. A 1-5 acre food plot is more difficult, but depending on its limitations, may be feasible. But the real question I always ask someone planning a food plot or just a garden in their back yard, why wouldn't you get one?? It is so easy, very cheap, and so important to your success that it doesn't make sense not to do one. I think that the advice you have received from others that have given it to you is pretty good advice.