robots.txt

robots.txt is a way to supposedly get search engines to leave your site's private resources alone, like a private API or something. But at the same time, you are making valuable resources easily known to attackers. It takes out the time and effort required for directory enumeration with DirBuster.

How can we come up with a better solution to robots.txt that doesn't paint obvious targets?

>>8
but then all someone has to do is use the same VPN as you (unless you mean a private/DIY one)

anyone here set up OpenVPN before? I was thinking about maybe doing my own VPN in AWS or some shit