I've found some nice sheds along fence lines. Seems like when the buck jumps the fence, the impact from landing will jar the rack off.
Your situation reminds me of that movie the "sandlot", where the kids hit the babe ruthe baseball in this mean mans yard and he had the meanest dog in the world named, "the beast". Watching them try to retreive that ball was hiiliaurs.

Bobby