Gregory's Server
@Em0nM4stodon are you aware of a way to keep an updated version of wikipedia locally? having a one time snapshot and only needing 110 gig is pretty rad (and i have the space, so i can trivially do this), but it would be nice to have a version that is updated, so if wikipedia ever goes offline, i'd in theory have the most recent updates and data available
|
7 comments
 @mttaggart @Viss @Em0nM4stodon I’ll third this. Just setting this up for a few site targets and I’d be interested in the delta sync and versioning (if it suddenly starts being vandalised or redacted)  @SecurityWriter @mttaggart @Viss @Em0nM4stodon I had a very quick look. There's torrents. I'd need to check whether they're for compressed or uncompressed contents - easier would be uncompressed. I'd set it up on zfs or btrfs, snapshot once one version is complete, then download updated version on top - unchanged files should stay the same, changed should detect they're different and get overwritten. Ideal would be an rsync mirror, or in some VCS @mttaggart @Viss @Em0nM4stodon its been decreasing for years.. calling Wikipedia a treasure at this point is a stretch far  @BigG @Viss @Em0nM4stodon Since the infrastructure is already in place to host/seed the torrent, and do the scraping/packaging, I'd suggest your better bet instead of downloading/scraping it yourself is get involved in the Kiwix project that creates that 110Gb zim - they used to be updated regularly, but the latest is the Jan 2024 one; who knows maybe that team is days away from releasing the 2025.01 zim and/or they just need some help? @Viss @Em0nM4stodon seems like they used to update more regularly but their downloader broke last year. https://social.treehouse.systems/@Pokemod97/113873646281739946 |
@Viss @Em0nM4stodon I am also interested in this, but recognize that information quality may in fact decrease with time at this point.