@derekwillis @ProPublica 10k accounts, 10 tweets/day, 100k tweets/day, what, 1k of data per tweet, less than 500 GB over 10 years, I feel like it wouldn't more than $100/mo in cloud services, would it? I don't have any sense of the budget available for something like this, though.
@JetForMe @ProPublica it's complicated, Politwoops relies on the streaming API, which means it has to run 24/7, because in order to know which tweets have been deleted you have to have copies of them first (the notification is just the ID).
That latter bit is what's currently broken - Twitter isn't sending those notifications. So we would have to, I guess, go back and check if tweets had been deleted, which is possible but you lose time context.
Also, ppl delete stuff after a long time.