I told him over the phone that i was gonna take that sign down and he said "you can't do that, you gave it to me".
You gave him the sign just like he gave you the rights to hunt. He knew all along that he was sending people into the woods during the hunting season in effect ending your hunting. Take the sign and don't look back.