Email or username:

Password:

Forgot your password?
Neil Gaiman

Is there an app or an easy way to harvest, download, etc your Twitter DMs? I'm downloading my Twitter data but I'm realising there are conversations etc with people going back to 2008 in Twitter DMs I'd love to be able to access if the site goes away. When you download Twitter Data it gives you your DMs... but with no information on who you were talking to. It looks like this, and is thus a very special kind of useless:

66 comments
joshua byrd 🦄

@neilhimself there are tools that transform the raw data into browseable content. Will have a look for you when I get out of bed :)

joshua byrd 🦄

@mom101 @neilhimself this was the tool I was thinking of, but it looks like DM parsing is not implemented yet. Hopefully soon! github.com/timhutton/twitter-a

Press Rouch

@phocks @mom101 @neilhimself I've been plugging away at a different issue on this (parsing likes, because my Twitter account was almost purely a content ingestion machine). As part of this I have discovered that it is possible to get a guest token for the Twitter API, bypassing the need to sign up for a developer account. I might take a crack at DM identification next - however, I don't have any sample data yet (no DMs in my personal account, now waiting on the archive for another one).

Stephen Bannasch (316 ppm)

@phocks @mom101 @neilhimself

This capability is n active development right now. Look at the most recent comments on this github issue: github.com/timhutton/twitter-a

✨Krysalia ✨

@phocks @neilhimself it is browseable at the moment. The problem is that the recipient data is purposedly not incuded. No names nor avatars nor handles.

Bjørn Larssen 🇺🇦 🏳️‍🌈

@neilhimself Oh crap, I am downloading my data with the express purpose of knowing who I was DMing with :/

mariana :coentro: :cuscuzin:

@neilhimself I'd like to know that too. I was so bummed when I looked at my DM data

Stever Robbins

@neilhimself have you looked at TweetDeleter.COM ? I am not sure they handle DMs but I seem to recall their plan allowing you to keep an up-to-date archive. If they include DMs, maybe they tie the identity back to a handle via the API

Zuri

@neilhimself There's definitely something floating around for this on Mastodon

Lizzie OwlyCat

@neilhimself if you find out, could you please share your wisdom? Because this looks rather useless indeed...

Sebu

@neilhimself if you live in EU, a GDOR request should do it

Mira Kamdar

@neilhimself I deleted all my Twitter DMs going back to 2009, retaining any communicated contact information. My DMs included conversations with human rights activists, dissidents, journalists. I decided it was best to erase it as far as it was possible for me to do.

Joseph k

@mirakamdar @neilhimself wow I understand why you did that but it's horrible that you had to do it. Keep up the effort and thank you for being aware of potential vulnerabilities... So sad...

✨Krysalia ✨

@mirakamdar @neilhimself i'm sorry to say that you did not erase a thing.
If you try to erase, it deletes it "only for you" as stated on twitter. The recipient copy is intact. Try with two accounts, you'll see.

Lilian Edwards

@neilhimself i bookmarked this github.com/timhutton/twitter-a which is indeed one of these things for turning the raw download data into something more useful but not sure its exactly it. I dont know why the recipient isn't part yr data - redaction? Wld you get it w a GDPR request? @tnhh @mikarv? Not that I expect Twitter to do much abt them now

MiDWaN

@neilhimself Not currently, from what I've seen (I've checked all the tools available until now). Most handle only your tweets. The problem is, to get the DM info translated from the userId to a name, you'll probably need access to the Twitter API, which is an extra hassle (you have to sign up as a developer, get an API key, use that for your tool - but you cannot share it!).

BoulderBelle

@neilhimself I would be interested to know that as well!

Marco Ciappelli🎙️✨:verified: :donor:

@neilhimself I just boosted your post in the Infosec.exchange server… I am sure someone can help you there 😬
I could use that advice as well.

DaySleeper 💚

@neilhimself Can you open them and download each one seperately? I know it's a ridiculous amount of work but simultanously you could also decide which ones actually to keep. 🤔

Pyjama Consultant 🇮🇪🇫🇮🏳️‍🌈

@neilhimself I don't know if this works anymore because of the new CEO's view on "blotterware" but by request they should give it to you. Might take a few weeks instead of hours. Hope it helps.

You will get it in a chunk, and you have to do the archiving yourself.

help.twitter.com/en/managing-y

SoyDelBierzo :verified_paw:

@neilhimself looking for something that works.

BTW media is not downloaded, the data file only links to uploaded files in twitter.

You need this tool to parse data file and don't loose your images, etc

github.com/timhutton/twitter-a

voronoi potato

@neilhimself
I suppose you could always go through the dms and send each of them a message with their username so that you have it when you back up. Either way associating users and dms should be the kind of thing a script kiddie could whip up in a couple hours. I know twitter broke third party dm backup tools that used to exist but if you already have a backup you just need to add a username. I'll look around.

Brian Moakley

@neilhimself "a very special kind of useless" ... words reserved for anniversaries and Twitter purchasers

John Abbe (aka Slow)

@neilhimself I see it on the todo list (with a high priority if that's what the "#6" indicates) for this project. github.com/timhutton/twitter-a

Robert Vinet 🍁

@neilhimself Does Twitter now own our DMs? Or is there a way to download them that is not obvious to me?

Ryan Anderson

It looks like it stores the user IDs, not the "@name" values. Mapping it to even a Twitter handle is impossible without the mapping data on twitter's servers.

Pretty useless, but I understand why they need to use an immutable value like that since people can and do change their handle.

ASmellyOgre

@neilhimself somebody on a forum I frequent made a script recently to solve this very issue. It does require a Twitter API key, though.

knockout.chat/thread/43145/16#

Thomas Hurst

@neilhimself It's the same with your list of follows/followers - all it includes is a numeric user ID, no account information. Completely ridiculous.

Writing a script to retrieve the data should be simple enough, but you'd need to sign up at developer.twitter.com to get an API key so you could use it.

WeiserAlterElf 🎃

@Freaky @neilhimself Well, to me (just reading this, not having the problem, never sold my soul to the birdsite) this sounds somewhat like people telling me years ago "using nerdy selfhosted opensource platforms is meh, we better use something readymade for free..." and now having to learn to write your own code urgently for the sole purpose to get a copy of the data which you handed over to that 'free' service, which hardly has employees left? Sorry... not sorry.

Natanael ⚠️

@neilhimself semiphemeral I think will both let you retrieve and download it and then delete it from the account

Melon Husk (parody)

@neilhimself Just an addition to the existing replies that correctly note the numeric user id is in the data, you don't actually need an API key to translate the id into a name, as long as this continues to work:

twitter.com/intent/user?user_i<USER_ID>

✨Krysalia ✨

@neilhimself i don't know if you'll get answers for this but i'm really interested in such a solution too.

At least, if there would be a way to tag those conversations in the archive, with who twitter says it is when we open it directly on twitter ><...
We can do that by replying to each convo with the name of the recipient but it means to wake up some old/nasty ghosts, in the process of sending them a new MP ><...

Micah Pollak ☃️

@neilhimself I believe they download as part of everything else if you use the "Request Twitter Archive" feature in your Twitter account settings. Although, I've been hearing mixed feedback about if that feature even works still (it did for me two weeks ago).

Neil Gaiman

@pollak they But look at the picture tonsee what downloads.

Micah Pollak ☃️

@neilhimself Ooh, my mistake, I see what you're saying. You have the content but the identities of those you're DM'ing are missing.

If you open the downloaded archive file itself you can get the "senderID" (an ID number) for each of the DMs and then can look up that ID on a site like TweeterID (tweeterid.com/) to see who that is.

There should be an easy way to automate it, but I don't know if anyone has yet.

Dark Days-i

@neilhimself there have been a number of suggestions of an app called Debirdify to assist with finding your Twitter peeps on Mastodon. I'm not sure whether it will capture your DMs, though.

Dinah 🕊🇺🇦

@Astrobeej @neilhimself Looks like that part is done, but it doesn't bring along DMs in a useful way.

I'm not sure how many DMs you could get showing at once in a web page by scrolling down, but you could "Save As..." in the browser to keep a copy of that list of people you messaged with. As viewed on screen it doesn't show the full name, but looks like (from search in page behavior) that it probably would be there in the HTML.

Michael K Johnson

@neilhimself I don't see anywhere in the data that it mentions any names. It has IDs and URLs embedded, but those depend on twitter continuing to function. I did not just point a browser at the HTML file, I went looking through the whole thing and found only the numeric IDs and userLink URLs that point back to twitter.

😭​

FromLefcourt

@neilhimself I heard for $8/month Twitter will be adding this as their "reverse verification" system. Instead of paying to verify who you are, everyone else will have to pay to know who they've been talking to.

Christian Ströbele

@neilhimself Should be possible to parse ids from dm archive to account names, as github.com/swedishmike/twitter attempts, or directly via browser emulation, but riskier, see github.com/Mincka/DMArchiver (haven't tested both though)

Christian Ströbele

@neilhimself Should be possible to parse ids from dm archive to account names, as github.com/swedishmike/twitter attempts, or directly via browser emulation, but riskier, see github.com/Mincka/DMArchiver (haven't tested both though)

Elika Kohen

@neilhimself - I am not sure when Mastodon will have a tool to import posts, but importing DMs is nigh impossible unless the @names are identical. It may be a good idea to backup your Twitter posts and messages using their own tools. I highly doubt they will shut down Twitter. They might rebrand it or resell it first. Here is a reference on how to download your own Twitter data: help.twitter.com/en/managing-y

Neil Gaiman

@elikakohen see the picture on my original post on what you get.

ethang

@neilhimself Honestly, why not just let it go? 2008? Come on man...

Neil Gaiman

@eglove because there are hundreds of DMs over the last 14 years many of them with email addresses and phone numbers in them. But if I don't know who the sender was they are all useless.

gz

@neilhimself I wonder how long before Twitter crashes due to vast numbers of data downloads.

Lady Zombie

@neilhimself literally trying to figure out the same exact thing

Torgny Bjers

@neilhimself That looks like an oversight in the export for sure. Do those accounts still exist on Twitter? If so, then that's most likely a bug!

TimmyK

@neilhimself Sounds like you just have to have an eccentric memory for dates!

Neil Gaiman

@Bombadil665 no, nothing in there on attaching IDs to the people in DMs.

FaillaDrum

@TenTonHeart
🔆 love from failladrum in these new streets.

Liz Durano

@neilhimself I used the Tweetdeck app to have my conversations (DMs) sent to my email.

Go Up