progrider indexed frequently by google

Do we want that?

Admin, can we get a ROBOTS.TXT?

And how am I supposed to search for my old posts? Fuck you nigger.

should /prog/ become a closed group? Then we could get shits and giggles by leaking threads.

By storing them locally >>3-baka.

I don't get why people think it's okay to be subject to opt-out things when web crawling should really be opt-in instead.
Also, robots.txt is bad for security purposes:
http://www.theregister.co.uk/2015/05/19/robotstxt/
And what if robots.txt is someone else's opt-in?
Opt-outs are dumb, especially because they overlap and conflict each other in many cases.
If you want to opt out of Google's network mapping, you have to add "_nomap" to the end of your wireless network's SSID. But what if someone else comes along and says that they're also doing an automated wireless network info slurp and want you to add "_optout" to the end? You can't have both opt-out strings at the end, because there's only one end. If you do SSID_optout_nomap only _nomap will get recognized. If you do SSID_nomap_optout only _optout will get recognized. And what if there are so many opt-outs that it goes beyond the SSID limitation of 32 characters? What if your SSID has to comply with corporate naming conventions and it's already close to 32 characters? Then do you have to omit some of the identifying text to put in the opt-out? Eventually will it get to the point where your entire SSID is nothing but opt-outs? Additionally, what if other search providers or web crawling services want you to add your own something.txt file to your server? How horribly will this scale? Will we see people with gigabytes upon gigabytes of opt-out text files just so they aren't subject to snooping from evil corporations that want to sell advertisements next to your stolen intellectual property?
What next? Should I write robot_optout on my door so that drones won't try to break it open and take pictures of the inside of my house for Google House View?
This is fucking insane.

Robots.txt can be good for copyright.

Robots.txt! Huh! What is it good for? Absolutely nothing!

le sekrit club xd

>>6
The Internet's supposed to be a PUBLIC place. It's very simple to prove that you don't want your website to be public: hide your website behind a login page.

>>10
There's a difference between having a public site that anyone can visit and having a site that is crawled by search engines that do nefarious things with your data and display advertisements alongside links and text related to your web pages (essentially making money off of your intellectual property and shoehorning your content in with certain keywords and displaying it next to who knows what kinds of other sites).

>>11

having a site that is crawled by search engines that do nefarious things with your data and display advertisements alongside links and text related to your web pages (essentially making money off of your intellectual property and shoehorning your content in with certain keywords and displaying it next to who knows what kinds of other sites).

That's life! Hide your super special work behind a login page. If you want your website to be public, then it's going to be public with every consequence of being public.

>>2
Do we really want Google users in here? It's bad enough that we have people posting Reddit and YouTube.

>>11

display advertisements alongside links and text related to your web pages

I'm pretty sure no sane company would rejoice seeing their ads near your gay porn riddled hole.

>>9
E P I C
P
I
C