twXplorer — A smarter way to search Twitter

TwXplorer, a new social-media research tool launched today by the Knight Lab, started with one journalist who told us he had a problem.

Peter Slevin, a Medill faculty member, has been working on a book about Michelle Obama. As part of that work, he periodically tracks her place in the "global conversation" by searching Twitter for references to the first lady.

What he gets back: a long list of tweets mentioning Michelle Obama. He can do little more than scroll through them, jot down notes about what he finds and tweak his search terms. As a way of keeping tabs on what people are saying about the first lady, it isn't very effective — or efficient.

"The problem when searching Twitter for a very common term such as 'Michelle Obama' is that there are few if any filters on what you get back," Slevin said. "Especially in breaking news situations, you find yourself scrolling through a very long and undifferentiated list of tweets."

Slevin's problem inspired the first iteration of twXplorer this spring, the work of a team of journalism and computer science students in a class led by me and Larry Birnbaum of the Knight Lab faculty.  Over the summer, the Lab's staff built the production version.

TwXplorer adds value to Twitter searches in different ways.  Here are the key things you can do with twXplorer:

See search results four ways

For any search terms you enter into twXplorer, you get four different ways to see your search results:

  1. Up to 500 recent tweets containing the terms you entered.
  2. In tweets that include your search terms, a bar graph showing the most popular other words that appear.
  3. The most popular hashtags included in tweets containing your search terms
  4. The most popular links in tweets containing your search terms.


Understand what Twitter users are talking about

At its core, twXplorer is a tool for searching Twitter in order to understand the global conversation about a topic. TwXplorer improves on Twitter search (and other search tools such as Topsy) by displaying the bar graph of the most common words and phrases used on Twitter in addition to whatever it was you searched for. In counting the most popular terms, twXplorer groups together any terms that have the same word stems -- for instance, "president" and "presidency" are combined. Add it all up and you get a good, visual overview of what people are saying about your topic.

Discover unexpected, but relevant content

The links section of twXplorer is a great way to find news coverage and commentary related to your search terms. For instance, the class team at one point searched for "New York bike sharing." They discovered articles about how useful the city's new bike-sharing program would be for tourists -- a topic they hadn't thought about.

Find good hashtags to follow or add to your tweets

The list of most popular hashtags could be useful to you in at least a couple of ways. First, you might discover a hashtag that you'd like to keep track of regularly.  To maximize the audience for a planned tweet of your own, you might find a good hashtag or two to add to it.

Refine your search through a "drill-down" approach

It's well-established that most people who use a search engine type in only a word or two -- even though longer queries will often produce more relevant results. TwXplorer has a clever approach for helping you refine your search: you simply click on a term or hashtag on the search results page and you see information only for the subset of tweets that include that term. This is an easy way to refine your search to find exactly what you're looking for. I, for one, would love to see a similar feature on other kinds of search tools -- even Google's web search.

Save your searches

Journalists -- and others -- often are interested in understanding what's being said on Twitter at different points in time. Slevin, for instance, wanted to be able to go back in time to see what Twitter users were saying about Michele Obama in the past. TwXplorer allows you to save your search -- capturing all four views of your search results -- any time you want to. Then you can go back and explore those results, drilling down to refine your searches. You can also delete saved searches that are no longer useful for you.

See hot topics on your Twitter lists

Once we realized how useful twXplorer could be, we looked for other ways to apply its technology within the limits of Twitter's Application Programming Interface. We discovered that we could apply the basic twXplorer search approach not only to recent tweets, but to the latest tweets being posted by members of any Twitter lists you have created or subscribed to. If you use lists to collect tweets from Twitter users you are interested in, this feature can be incredibly helpful. For instance, I have created a "hackshackers" list consisting of journalists who do computer programming and data analysis. But I scan tweets from this list only rarely. Using twXplorer, at any given time, I can see what the hot topics are among these Twitter users.

* * *

The first version of twXplorer was built by two undergraduate computer science students (Jeanette Huang and Allen Zeng) and journalism master's student Miguel Huerta.  They formed one of eight teams in the most recent of our "collaborative innovation" classes, which are a great platform to test new software ideas, to generate creative solutions and to involve students in the development of new tools for journalists, publishers and media consumers.

"It was one of the most successful projects in the spring class," Birnbaum said. "We could immediately see the value for journalists and other users, and it was clear that the Knight Lab could launch it for public use in a reasonable time frame."

Working with Larry and me, Knight Lab developer Jennifer Wilson built the new version of twXplorer this summer, with help from the Lab's design/research fellow Jessica Soberman (MSJ 2013) and art director Aaron Salmon.

You might be interested in knowing a little more about how twXplorer works. Here's an overview:

* OAuth sign-in: Before using twXplorer, you sign in with your Twitter ID. This has several implications. First, it means Twitter will identify your search request as coming from you rather than twXplorer as a website, which means many people can use twXplorer concurrently without fear of running up against the limits of Twitter's API. Second, it means we can access your Twitter lists for the twXplorer lists feature. And finally, it means we can associate saved searches with your Twitter account rather than making you create a separate twXplorer login.

* Search by language: By default, the twXplorer search looks for tweets in the language associated with your Twitter profile. But you can search for tweets in 12 languages. Twitter offers "best-effort" language detection, which is not perfect but can help you find tweets written in those languages.

* Find up to 500 tweets: To provide a relatively swift response, and comply with Twitter's API limits, twXplorer finds the 500 most recent tweets that include your search terms, then does not display those that Twitter codes as retweets ("new style" retweets as opposed to those where the content is preceded by RT). If twXplorer reports finding 400 tweets, it means it found 400 unique tweets -- which you can scroll through -- and 100 "new style" retweets.

* Zero in on the most relevant terms: TwXplorer excludes common words like "the." Then it looks not only for single words, but also user mentions (such as @KnightLab) and hashtag text (#chicago counts as "chicago"). It also looks for "bigrams" (two-word phrases) that show up more than once. If a bigram is common (say, "white house"), twXplorer doesn't count them again as separate terms.  TwXplorer groups terms together when they have a common stem ("look," "looks" and "looking" are counted as the same term).

* Count the terms: The number that appears next to any term, hashtag or link is the number of tweets that include that term. The counts include terms used in retweets, although the retweets are not all displayed.

* Drill down: When you click on any term, hashtag or link in your search result, twXplorer returns only the subset of search results containing the term you clicked on. If you click on a second term, the subset of tweets is narrowed even further. For instance, if you filter separately by "chicago" and "bulls," twXplorer will display search results only for tweets containing both terms.

About the author

Rich Gordon

Professor and Director of Digital Innovation

Journalism/tech intersection, my passion for 25 years, data journalism, Miami Herald web director, now hacker journalism.

Latest Posts

  • A Google Spreadsheets change affecting TimelineJS users

    Google recently changed something about their Sheets service which is causing many people to run into an error when they are making a new timeline. Note: there should be no impact on existing timelines! After this change, many of you click on the "preview" and get this message: An unexpected error occurred trying to read your spreadsheet data [SyntaxError] Timeline configuration has no events. There is a straightforward work-around, but it requires those of you who have...

    Continue Reading

  • How Americans think and feel about gun violence

    A man killed his wife, then himself. I want you to see his face and learn that he enjoyed fishing with his grandchildren. A small-time drug dealer is shot by two men in a parking lot. I find his Facebook profile and a photo shows him striking a playfully irreverent pose, giving the camera the middle finger. The photo’s comments take a mournful turn after a certain date. “Rest easy bro ???” Gun Memorial runs...

    Continue Reading

  • Software developers interested in journalism: Northwestern and The Washington Post want you!

    Northwestern University and The Washington Post are offering a unique opportunity for two talented software developers interested in applying their programming skills in media and journalism. Here’s the proposition: (1) a full-tuition scholarship to earn a master’s degree in journalism at Northwestern University, followed by (2) a six-month paid internship with The Post’s world-class engineering team, with the possibility of subsequent full-time employment. These opportunities are made possible by the John S. and James L....

    Continue Reading

  • What happened when Gun Memorial let anyone contribute directly to victim profiles

    If you’re reporting local or niche news, there’s a good chance that your audience collectively knows more about the story than you do. That’s especially true for us at Gun Memorial, a small publication with a nationwide mission of covering every American who is shot dead. In our latest, mostly successful, experiment, we let readers add to our stories without editor intervention. This article shares some lessons from that experience. Asking for reader contributions A...

    Continue Reading

  • How conversational interfaces make the internet more accessible for everyone

    This story is part of a series on bringing the journalism we produce to as many people as possible, regardless of language, access to technology, or physical capability. Find the series introduction, as well as a list of published stories here. In 2004, human-computer interaction professor Alan Dix published the third edition of Human-Computer Interaction along with his colleagues, Janet Finley, Gregory Abowd, and Russell Beale. In a chapter called “The Interaction,” the authors wrote...

    Continue Reading

  • Three tools to help you make colorblind-friendly graphics

    This story is part of a series on bringing the journalism we produce to as many people as possible, regardless of language, access to technology, or physical capability. Find the series introduction, as well as a list of published stories here. I am one of the 8% of men of Northern European descent who suffers from red-green colorblindness. Specifically, I have a mild case of protanopia (also called protanomaly), which means that my eyes lack...

    Continue Reading

Storytelling Tools

We build easy-to-use tools that can help you tell better stories.

View More