Usually, if the previous years rubs were sufficient enough to damage the tree, you will see a dark area on the tree where a rub would normally occur. Also, many times the bark will still be damaged and is normally very dark. This has been my experience.

Hunt the thickets