One notable difference is that sites that allow user-submitted content (e.g. this one) could be exempted by the user, whereas robots.txt et al require the server admin to intervene. (But I agree that this would be a feature and not a bug.)
robots.txt
One notable difference is that sites that allow user-submitted content (e.g. this one) could be exempted by the user, whereas
robots.txtet al require the server admin to intervene. (But I agree that this would be a feature and not a bug.)