You can only push a pure lead bullet so hard before you will very likely get leading in your bore. My guess is that your trajectory gain by going to lighter bullet isn't going to be that great, taking into account the leading issue and the speed limit it will impose. Those 460 grainers should make a very good choice for your "varmint rifle".