@msteenhagen let’s hope @internetarchive will still be around in a decade… but, yeah, electronic stuff is ephemeral in nature

@msteenhagen Well, old books linked to other books that are now also lost, so it's not really new.

@msteenhagen Film negatives are also a pretty good storage medium for visual things!

I think it shouldn't be underestimated when it comes to archiving things :3

@msteenhagen You know the waybackmachine, right?
http://web.archive.org/

(Big but inevitable drawback: anything archived that is behind a password is not accessible)

Basically, if information keepers and information providers do not implement persistent identifiers, we're at a loss.

@msteenhagen agreed. It’s even worse than just rot within domains https://mastodon.social/@urlyman/111244563848075412

@msteenhagen From 1996-2016 I saw about 20% link rot per year on a website with hundreds of external links. Even if half of them just moved, it was a big job to manage the links.

@msteenhagen ...that's why at least in government we have laws with regard to archiving , we also have a national archive etc. people are working hard there to archive information durable

@msteenhagen certainly I can think of a technical solution to the problem but it needs adoption and also acceptance to the fact that it won't drive any ad income 🤷‍♀️

@msteenhagen When Reddit shot itself in the foot, that killed a LOT of links. Many articles linked by search engines are deleted or missing. Also, social media entities adding login walls doesn't help.

It's all very temporary.

@msteenhagen One route might be something like this, alongside the Internet Archive:

https://zenodo.org/doi/10.5281/zenodo.10119195

@msteenhagen One would need to run https://perma.cc/ or https://archive-it.org/ recursively for every cited URL: it gets expensive quick. (And requires the #InternetArchive not to be burnt down by the knowledge arsonists aka big publishers, or you need to separately deposit the WARC files and serve them somehow.)

#digipres

@msteenhagen I have a bookmark folder with interesting articles that go back to 2005, and indeed a whole lot of the links are dead. So at some point I started to also print to PDF every time I save something there, so I have an archive of it all.

When I worked in academia, I used Zotero for references, which also saves webpages. But I stopped paying for storage when I left academia. I have a local copy of that archive too.

@msteenhagen That’s why I use Zotero as a bookmark tool. It downloads copies of web pages into my own private archive.
The sad design error of the WWW is that links do not exist. What we call “links” are mere pointers. Maybe Ted Nelson should have put a little more effort into his #hypertext system Xanadu in the sixties. https://www.wired.com/1995/06/xanadu/

@msteenhagen@provo.lol i run a little youtube archiver and it's downright scary how much stuff just... disappears, without any reason behind it

@msteenhagen

The digital world and the internet are volatile… just as human thought and opinion nowadays it seems. Attention span is eroding.

It is worrying indeed, we are our history, if we do not remember that, we will be repeating mistakes of the past…

@msteenhagen I am using ArchiveBox https://archivebox.io/ for years and return to it from time to time when looking for something, but this is a small private instance.

@msteenhagen better archiving legislation helps. And more responsibility from content delivery organisations

@msteenhagen To combat link rot as part of my workflow, every link I save to Pocket gets picked up by a nightly script and submitted to archive.org.

But it still depends on a single organisation, with an uncertain long term future, so it's not a great solution per se - though it's a convenient one for now.

I need to look if someone ever created a self-hosted archive.org type thing so I can save content locally too.

@msteenhagen
Also a concern and criticism in respect of streaming only media and licenced digital books.

Are we in an ege of ephemera?

@msteenhagen thousands of years from now, archaeologists digging thru the remnants of our society will find no durable artifacts other than plastic shells, and no literature. They'll probably conclude it was a cultural Dark Age. And probably be correct.

@msteenhagen@provo.lo l https://archive.org/

@msteenhagen One way is to try and get folks to maintain older versions of their sites.

It’s trivial to do if one cares at all for that sort of thing. But I have a feeling the issue is with the qualifier in that statement. In a world of disposable “startups”, I don’t think many people do.

https://4042307.org

@msteenhagen It’s OK. Google is sending all of your data to brokers that will inevitably get hacked making certain that all of your data is immortalized on hard drives across the globe.

@msteenhagen Similar issues with digital book formats. Amazon was a proponent for .mobi format but have now abandoned it in favor of EPUB. OEBPS readers are hard to find. Adobe's DRM can lock access to titles making them unusable. PDF has a ton of accessibility issues.

Digital publishing has made the *dissemination* of information fast easy and cheap. But *archiving* that information is not only fraught with digital issues but legal ones like DMCA. We need to rebalance this equation.

@msteenhagen it is by no means a solved problem, but in the legal profession, we've been moving more and more to the use of archives administered by relevant libraries. See e.g. https://perma.cc/libraries In this way the same institutions (law schools) that publish our scholarship (law journal articles) can also house relevant archives.

@msteenhagen
https://archive.org/web/ ?

@msteenhagen
My father was a photographer. When he died I inherited suitcases full of photographs, some are over 100 years old. I have a digital camera, I have taken thousands of photographs. When I die no one will see them ever again.

@msteenhagen it gets worse. Websites 5-10 years ago would at least lend themselves to being archived — either in services like Wallabag, or just by saving the HTML and resources using wget or some spider.

Websites today, with their JS overload and dynamic content, and CloudFlare "stopping bots", quite often do not allow themselves to be archived.

It's not just link rot, it's also the rot of our ability to archive stuff at all.

@msteenhagen I have been struggling to source a quote for a small article that was only ever shared in a YouTube video once. No where using it as their source even mentions what the name of the video even is.

Folks don't even know how to cite things to even help folks find where they might be stored away anymore and with things vanishing so fast we're going to lose important knowledge.

@msteenhagen It is a major problem… does anybody else think that the term „link rot“ is bad, since it applies that only pointers are disappearing, not the content? It should be „content rot“.

@msteenhagen I know of projects that mirror entire websites, so they could still be accessible of the main site goes away. I also personally, ocasionally, download specific webpages that I like, so as long as I have a browser I can still read that particular snapshot

@msteenhagen
although imperfect, I feel @internetarchive is so far the best existing attempt to address this problem. The thing is - the scope of its ability to handle link rot is sharply limited to what people submit snapshots of.

@msteenhagen The only reason we know anything at all about ancient peoples is because they wrote things down on stone, which is virtually indestructible.

@msteenhagen All good points, and a major concern.

I had the pleasure for several years of contributing my organization's publications to a "digital repository" at a major university library. The folks running the repository and their bosses who made an institutional commitment to it are unsung heroes. But if I had a time machine it would be interesting to know how long that commitment lasts. I'd give good odds to 50 years, about the span of a long professional career. But after that?

@msteenhagen

Now take it one step further: any of your life’s work, insights and expression, that you store in a digital format is likewise ephemeral. At the same time, storing all that information in The Cloud uses huge amounts of energy and water, polluting air, water and the atmosphere.

@msteenhagen
One of the busier pages on my website is a mirror of the site for a circa-2002 mp3 tagging program, because that version of the software and documentation are still used, but no longer exist other places like the original domain.

@msteenhagen check out “Robustifying Links to Combat Reference Rot” https://journal.code4lib.org/articles/15509 / cc @martinklein @shawnmjones

@msteenhagen National archives have actually started to take notice and archive (depending on the country) content from various official and unofficial websites with the same or similar standards as they archive books, newspapers and other published articles.

https://en.wikipedia.org/wiki/List_of_Web_archiving_initiatives

@msteenhagen Bob Frankston, who invented the spreadsheet, has a good suggestion, Forever URLs.

https://rmf.vc/ieeeforeverurls

(Read it carefully, Bob selects every word and thought with care)

@msteenhagen I’ve always been wary of Tweet embeds in serious publications. Basically every news outlet has embedded tweets in a way that depends on those accounts, tweets, (and Twitter writ large) being online forever. It seems like a mistake! What if a syphilis-brained billionaire buys Twitter and breaks every previously functioning part of the company?

@msteenhagen

Sounds like you have your own solution, archive in print. Store in a library.

@msteenhagen

"What to do?"

Smaller fragments, and you save hashes of source and target with the link. That way, if you lose the big original chunk you can try to brute force a solution to the hash and reconstruct the information.

But that would require a fundamental change of how we use links and how easy it is to write them.

The web would need to be rethought as an archiving mechanism, not just a snapshot that may or may not work.

Or we start creating small personal archives.

@msteenhagen Only say: yes, it is

@msteenhagen

Someone should whip up an extension for this.

When a 404 is hit, it could look for the same page on the wayback machine and if available, direct the user to that one.

@msteenhagen this is why the internet archive is so vital, as are data hoarders

@msteenhagen
Indeed and format rot might be even worse. Will the pdf specification be the same 20 years from now?

Of will you have to go to Internet Archive and download some OS emulator to run today's PDF (or JPEG or .xls or whatever) reader?

@msteenhagen I have saved a set of browser bookmarks that is about 20 years old, mainly to reuse the folder structure. They are mostly in home repair, hiking, careers, and child raising. Only about 15% of the links still work. Another 10% can be found by searching the sites they were on or in a larger search.

@msteenhagen This is one of the (many) reasons why the lawsuit that "publishers" are bringing against the Internet Archive is so iniquitous.

@msteenhagen

There is the save page as option in browsers. A pity it is totally broken for most of modern sites. Most of them lost format, other even content. And it is hard to be sure you stored it the right way, since the saved page it is very likely depending on content on other servers. Testing it offline can help, of course, but you will need to empty your cache.

It's getting worse every decade.

@msteenhagen I once worked for a company where, whenever you used a link in an internal documentation, a PDF version of that website was automatically created.

@msteenhagen the term "link rot" suggests everyone who puts anything on the web is expected to keep it there forever. Why do folks expect that? Think of it as phone calls or chit-chat. Before 1990, *none* of that is preserved.

@msteenhagen

All is maya.

The "backup" is not the "archive".

These days I feel like Cassandra, having spent much of my career trying to warn of the very problems you describe. The corporatization and commercialization of the Web has led us to this situation and I honestly see no good solutions. Everything available to us at this point is "patchwork". I suspect what we will see is pockets of things that endure...

@msteenhagen I wonder if print archives of websites could be an option. I think that still isnt a great option, but it is less ephemeral than it all being a digital archive of a digital medium, it adds a physical layer to it

@msteenhagen you might be interested in:

Mohamed Aturban, Michael L. Nelson, and Michele C. Weigle, “Where Did the Web Archive Go?,”

The exact problem you mentioned!

@msteenhagen

This is well-known, and Roy T Fielding (author of HTTP spec, Apache web server pioneer, ++) warned about this in the 1990s, and promoted that all links was "dated", i.e.

http://hedhman.org/1998/12/my-info-page.html

and that one never removed old content, to preserve links forever.

Can't find the source, maybe it was just on a mailing list.

@msteenhagen almost like it was never a good idea to make everything primarily digital in the first place.

@msteenhagen

Agreed.
My website has been around for about 20 years and has lots of hot links. Every day more of them succumb to link rot. The best I can do is replace the dead links with equivalent links into the Internet Archive.

@msteenhagen Maybe we need a physical building that's just filled with tape drives of the archives

@msteenhagen and the Internet has destroyed the commercial incentive to publish factual books on paper.

@msteenhagen paper

@msteenhagen That's how planets get forgotten from galactic empires, copyright enforcement!
https://en.wikipedia.org/wiki/Hachette_v._Internet_Archive

@msteenhagen I've experienced this as well and it seems to be getting worse. I don't know, but is link rot a problem at sites such as the Internet Archive, Wikipedia, Library of Congress? I hope not. But as one commentor wrote, "electronic stuff is ephemeral." And that truth is indeed-- "terrifying."

@msteenhagen @blogdiva According to this https://www.protocol.com/internet-archive-preserving-future, the wayback machine alone is about 45PB, which isn’t that much in HDDs these days. About $1m. But servers, energy cost, as well as bandwidth—and most importantly maintenance—are not cheap. The legal risk is probably the biggest challenge, though. Do you delete everything archive dot org deletes when they get a DMCA complaint, or do you ignore it in order to preserve more?

@msteenhagen It takes strong institutions to pass the test of time. https://web.archive.org/ is the strongest we have at the moment.

@msteenhagen Get the Wayback Machine app for your phone (Android has it, not sure about iPhone). You just paste the link into the app and it goes and scans it and you get a link directly to the page saved on the Wayback Machine. I don't think it's going anywhere for a long time. Or is that the "archives" site you were talking about??

@msteenhagen @Et_Alia —

Repeat after me: "The backup is NOT the archive. The backup is NOT the archive..."

@msteenhagen I’ve got a 20 year blog and everything is dead, embeds are the worst but most links go away within a few years! Never mind 20!

@msteenhagen
One of the goals of #SafeNetwork is to solve this by storing public data forever.

@msteenhagen as a kid, if there was a particular article I'd wanna save I'd copy paste into a word document, or recipes I'd copy paste into my own digital notepad. Both are applications so as long as they themselves don't break, I can access when I want. Of course, applications breaking happen suddenly so perhaps not a great option either...

@msteenhagen

I have had this issue so many times. At one point, I started using Link Accessed on Specific Date as my guide. And save copies into my Scrivener. That wasn't always possible.

I also run a bookmarks check every few months. It isn't uncommon for several sites to have simply disappeared from the internet during that time.

@msteenhagen And yet the URLs on my non-commercial personal web site still work 25 years later. ;)

@msteenhagen this is what makes @internetarchive so damn important.

As a kid I remember being told "Be careful, any you out in the internet is there forever."

It sounded scary at the time, but now I realize that the alternative is even scarier.

#internetarchive #digitalpreservation #waybackmachine

@msteenhagen Store stuff on your hard drive or it will be gone. Heck store stuff on your hard drive and it might still be lost due to fuckups or hardware failures. Sadly, even archive.org (with their pretty awesome track record) might go down when funding dries up.

#archives #digitalhumanities

@msteenhagen This is incompetence on the part of the technical and/or content team. Every redesign, migration, or other major change I have been part of had redirects as a top priority.

On the better projects, we also had the resources to go in and update old links to minimise link rot.

There are no excuses not to do this.