Unless you're a technophile, you may never have heard of something called robots.txt. As David Pierce explains at the Verge, this simple plain-text file has for the last 30 years performed a not-so-simple mission: It has "kept the internet from chaos." He describes it as a "handshake" agreement among those who built the web, a "mini constitution for the internet, written in code." The text file allows those who create websites to determine who gets in—for example, which search engines can scrape your site—by governing the behavior of web crawlers. This was especially crucial in the early days—think the 1990s—when "all it took was a few robots overzealously downloading your pages for things to break." You can check it out at yourwebsitename.com/robots.txt.
"It's not a perfect system, but it works," writes Pierce. "Used to, anyway." The new problem, not surprisingly, revolves around artificial intelligence. Today's AI-powered web crawlers are nothing like their relatively primitive ancestors. They're far more powerful and secretive as they scour the web to "build massive sets of training data," writes Pierce. As a result, "the fundamental agreement behind robots.txt, and the web as a whole—which for so long amounted to 'everybody just be cool'—may not be able to keep up either." Thirty years ago, the "humble plain-text file" was good enough to keep things under control within a far more innocent internet. Today, it's "starting to look a little old-fashioned." Read the full story, which explores some potential remedies. (Or check out other longform recaps.)
,
uBlock 
,
and
Ghostery 
)
