I don't get why people think it's okay to be subject to opt-out things when web crawling should really be opt-in instead.
Also, robots.txt is bad for security purposes:
http://www.theregister.co.uk/2015/05/19/robotstxt/And what if robots.txt is someone else's opt-in?
Opt-outs are dumb, especially because they overlap and conflict each other in many cases.
If you want to opt out of Google's network mapping, you have to add "_nomap" to the end of your wireless network's SSID. But what if someone else comes along and says that they're also doing an automated wireless network info slurp and want you to add "_optout" to the end? You can't have both opt-out strings at the end, because there's only one end. If you do SSID_optout_nomap only _nomap will get recognized. If you do SSID_nomap_optout only _optout will get recognized. And what if there are so many opt-outs that it goes beyond the SSID limitation of 32 characters? What if your SSID has to comply with corporate naming conventions and it's already close to 32 characters? Then do you have to omit some of the identifying text to put in the opt-out? Eventually will it get to the point where your entire SSID is nothing but opt-outs? Additionally, what if other search providers or web crawling services want you to add your own something.txt file to your server? How horribly will this scale? Will we see people with gigabytes upon gigabytes of opt-out text files just so they aren't subject to snooping from evil corporations that want to sell advertisements next to your stolen intellectual property?
What next? Should I write robot_optout on my door so that drones won't try to break it open and take pictures of the inside of my house for Google House View?
This is fucking insane.