How the Wayback Machine is fighting linkrot

Link Rot

The phenomenon of link rot poses a significant threat to digital history, as large portions of the web become inaccessible over time. Nilay Patel highlights that 38% of links from 2013 are no longer accessible, representing a substantial loss of media knowledge and online culture 1. Mark Graham explains that various factors contribute to this decay, such as companies going out of business or changes in government policies leading to the shutdown of media organizations 2. He notes, "Thousands of local news organizations have shut down in the United States over the last ten or 15 years" 2.

   

Platform Shift

The transition from open to closed platforms and the rise of ephemeral content present new challenges for the Internet Archive's preservation efforts. Mark Graham discusses how the Wayback Machine provides a time-based view of URLs, capturing snapshots of the web that might otherwise be lost 3. However, the increasing use of web scraping by AI companies has made preservation more contentious, as many organizations now restrict access to their content 4. "All the generative AI companies are scraping the entire web and using it to train their LLMs," Nilay Patel notes, highlighting the tension between preservation and proprietary interests 4.

   

Closed Platforms

Archiving content on closed platforms presents significant difficulties, as these environments often limit access to culturally or historically significant data. Mark Graham points out that platforms like Instagram and Facebook restrict access unless users have an ID and password, complicating efforts to preserve content 5. The debate over web scraping further complicates this issue, as many websites now block automated tools that are vital for preservation 6. "Lots of websites are now making changes to block these scrapers," Nilay Patel explains, underscoring the evolving landscape of digital archiving 6.