Twitter Archeology

The amazing thing about Twitter, and its user-created hashtags, is that it lets everyone see what the hot topic is right now, this very instant, and maybe the instant right after this one. But how good is it at archiving those conversations so we can examine the hot topic at some later date?

As part of the writing process for The New New News, Paul Mullin asked me to find “all the tweets about Maurice Clemmons.” So, just like Indy, I donned my Stetson hat, clipped my whip to my waistband and went out adventurin'.

First thing I did was log in to Twitter and type “Maurice Clemmons” and the associated hashtag #WAShooting into the search bar. No results. Next, acting on a tip from Paul, I scoped out the Library of Congress, who has a deal with Twitter to archive all of the tweets, ever. You can't search this archive online. I emailed the LOC help desk and they said that they can't search the archive, either. Because the archive isn't actually in their building yet.

Then I did other things for about three weeks.

Paul wondered what had happened, so he emailed me, and I dusted myself off and journeyed into Google. You can now get tweets in your search results, and they backlog them as far as they can.

Unfortunately for this project, the Google search results don't go back in time far enough to capture the moment, and they aren't comprehensive. So then I searched for #WAShooting.


"Was hooting." I didn't get it at first.

I scrolled a little further down the results page...


And I was led to a site called Twapper Keeper, which, as it turns out, does exactly what I need: it archives tweets, sorts by hashtags, and creates archive files that export as .csv. Perfect! But the only reason I found it is because @kenrufo was on the same archeological expedition, just a few days before me. If I had taken this trip in June, I would have come up empty-handed.

I now have 5,700 tweets containing #WAShooting. I'm guessing there's more out there, and certainly many thousands with the phrase “Maurice Clemmons.” This will have to do for now. Sort of interesting that this obscure little site did for me what the Library of Congress couldn't.