The text file that runs the internet

A new addition to my collection of in-depth profiles of the stuff our modern world runs on. If you own a website – be it the Weekly Filet, The New York Times or Wikipedia – a simple text file called robots.txt is your best option to tell the most powerful tech companies in the world what they can and cannot do with your content. Invented three decades ago, by people «who believed that the internet was a good place, filled with good people, who above all wanted the internet to be a good thing.»