It say's a lot about you that you have this concern! As others have said,smile,give a wave and back ou the way you came in.Then head some where else,hopefully some where that doesn't serve to cut off game movement that is headed in the direction of the hunter you saw.
It is unavoidable very often,if you treat other people in the same way you would like to be treated you will not have a problem.