For three decades, a tiny text file has kept the internet from chaos. This text file has no particular legal or technical authority, and it’s not even particularly complicated.
https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders