How the Wayback Machine is fighting linkrot

Storage Challenges

The Internet Archive faces significant data storage challenges as it manages an ever-growing collection of digital content. Mark Graham explains that the Wayback Machine processes more than a billion URLs daily, requiring extensive storage solutions like spinning disks and SSDs 1. This massive operation involves writing data to multiple locations for reliability, with the Archive purchasing over six hard drives daily to keep up with demand 1. The physical nature of digital preservation is highlighted by the Archive's use of various storage locations, including an old church in San Francisco and an abandoned coal mine in Norway 2.

We're talking about preserving a very digital, somewhat ephemeral medium on the Internet. And the actual process of it is extraordinarily physical.

--- Nilay Patel

Despite these challenges, the Archive remains committed to its mission of preserving digital history for future generations.

   

Preservation Techniques

The Internet Archive employs various preservation techniques to ensure digital content remains accessible over time. Mark Graham notes that the Archive's funding comes from a mix of program-related activities, donations, and support from high-net-worth individuals, allowing it to continue its mission 3. The Archive faces challenges like hyper-personalization and geopolitical boundaries, which complicate the archiving process 3. Despite these hurdles, the Archive actively works to capture ephemeral content from platforms like TikTok, using domain experts to prioritize culturally significant material 4.

Some of it is, yeah. Actually, TikTok is one of those platforms that we're doing a fair amount of archiving on.

--- Mark Graham

This approach ensures that even the most transient digital content is preserved for future analysis and understanding.

   

Archival Priorities

Prioritizing archival content is a complex task for the Internet Archive, as it balances cultural significance and technical accessibility. Mark Graham explains that the Archive respects rights holders by considering requests to exclude certain content, while also weighing public interest 5. The Archive collaborates with over a thousand partners, including librarians and infrastructure providers, to curate material for archiving 6. However, the rise of closed platforms like Instagram and TikTok presents challenges, as these environments often lack the open accessibility of traditional web content 6.

There's a growing body of information that lives on more closed platforms, even if they are exposed to the web.

--- Nilay Patel

The Archive's efforts to navigate these complexities highlight its dedication to preserving a comprehensive digital history.