User testing: how news designers and developers add context to quantitative data

Last week I wrote about how news organizations use A/B testing to help iterate on design elements such as page layout and headline writing-style in order to increase reader engagement. The technique provides essential information about what a reader is doing, but it does have limitations.

“When you’re only looking at metrics you see the what, but you don’t see the why,” said Steve Mulder, director of user experience and analytics at NPR Digital Services.

A/B tests lack context. In designing media products such as a homepage, story, or app, the motivations for user-behavior are as important as the behavior itself. Digital analytics may show that a reader is spending a lot of time on a story page, but not why. Is the story interesting? Confusing? Hard to navigate?

Observing people interact with a news product can help designers create better experiences for them. Observation can take various forms including eye-tracking and remote testing, but one-on-one observations are particularly exciting for newsrooms because they can be done quickly, inexpensively and they are informative.

Here’s how different news organizations have used one-on-one observations to improve specific products.

Washington Post

In the summer of 2012 The Washington Post launched a significantly updated version of its iPhone app. Before launching the new app, director of mobile Julia Beizer and her team watched readers use it.

It became clear that some of the new features were not intuitive to everyone. One particularly problematic feature was a customizable bookmarks bar.

The Washington Post added tool tips to its app to help readers understand new features.

To help readers figure out these new features, the team added “tool tips” screens (see image above). While Beizer acknowledges that these screens can be annoying and that not everyone might read them, they allowed the Post to keep and highlight a new app feature it was proud of.

Boston Globe mobile app

The Boston Globe saw a similar problem when designing its new app in June 2013, but found a different solution.

In an early beta interface of the mobile app (the screen on the left below) readers could pick which sections and in what order they would appear on their homepage. For example, if a reader only wanted to see arts and lifestyle, she could choose to display only those sections on the homepage and arrange their order (arts or lifestyle stories first).

User testing motivated the decision to eliminate sorting icons from the Boston Globe mobile app.

Testing showed that no one knew what the sorting icon (the horizontal lines stacked on top of each other) meant and that none of the users were re-sorting the sections, according to Damon Kiesow, senior product manager at the Boston Globe. Instead of adding a feature like explanatory text or tool tips screens, the Globe got rid of the problematic feature.

“UX testing through the development process is more about removing features and removing obstacles than adding features,” Kiesow said.

The process was simplified in the next iteration (the screen on the right) to eliminate this problem. Instead of allowing people to sort the homepage sections and the order that the stories appear in, readers could select which sections they wanted and the stories would automatically show up in reverse chronological order.

The homepage of the app went from being an editorially ordered page to a most-recent stream of stories. Kiesow believes that this was preferable for a particular audience accustomed to news feeds and updates from services like Twitter and Facebook, in which users are shown the latest information and not necessarily what they are most interested in.

NPR Apps

Brian Boyer’s team at NPR is working on relatively small projects that take around a week to build. While there is sometimes a rigorous usability test approach, it’s more often “off-the-cuff," he said. This could mean asking people walking by for help or going to someone’s office and having them test a project.

NPR’s Fire Forecast is a map app that “shows current large wildfires and forecast burning conditions in the lower 48 states.”

The NPR News Apps team went through several rounds of prototyping with Fire Forecast.

In early iterations (far left image) buttons were flat and the search box had no “go” button. Boyer and his team tested the app with different people and learned two important lessons.

First, it was evident that people were not interacting with the buttons, because they were not designed to look like actual buttons. In response, the team defaulted toward standard-looking controls that people recognized.

“The second we popped those buttons out, people immediately knew what to do,” Boyer said.

Second, the search box on a phone acts like a submit form in which you can type and then hit search. There was no obvious way to take action without a “go” box, so one was added in. These changes helped readers take full advantage of the app’s features.

What I’ve learned so far

Exposing products to users early on can help a news organization make sure that they are releasing and developing them with audience focus. One-on-one user-testing can help product development teams make informed decisions around what functions or designs to add or eliminate to aid in the larger mission of getting content to reader.

As Alex Wright, former director of user experience and product research at The New York Times explains, “It’s not really about testing the editorial content, but about figuring out the most effective way to deliver that content, and helping users find stories that are interesting to them.”

As evident from the examples above, the formality of the test doesn’t necessarily correlate to how informative it is. This means that whether you can afford to hire a firm to help test products, or if you can only dip into a local coffee shop to run through some task analysis with a willing participant, there’s valuable insight to be gained. After all, the NPR News Apps team didn’t need to hire a company to gain insight into the fact that people were having trouble figuring out how to conduct a search.

Last week I wrote that “design decisions can and should be rooted in data.” The quantitative data obtained from digital analytics works hand-in-hand with the qualitative data gathered through the kind of observational research and one-on-one testing that news organizations are doing. These two strategies for usability testing work together as on-going techniques for improving products.

When designing for the user-experience the why something is happening of qualitative data is just as important as the what that quantitative data shows.

About the author

Jessica Soberman

Graduate Fellow

Latest Posts

  • Prototyping Augmented Reality

    Something that really frustrates me is that, while I’m excited about the potential AR has for storytelling, I don’t feel like I have really great AR experiences that I can point people to. We know that AR is great for taking a selfie with a Pikachu and it’s pretty good at measuring spaces (as long as your room is really well lit and your phone is fully charged) but beyond that, we’re really still figuring...

    Continue Reading

  • Capturing the Soundfield: Recording Ambisonics for VR

    When building experiences in virtual reality we’re confronted with the challenge of mimicking how sounds hit us in the real world from all directions. One useful tool for us to attempt this mimicry is called a soundfield microphone. We tested one of these microphones to explore how audio plays into building immersive experiences for virtual reality. Approaching ambisonics with the soundfield microphone has become popular in development for VR particularly for 360 videos. With it,...

    Continue Reading

  • How to translate live-spoken human words into computer “truth”

    Our Knight Lab team spent three months in Winter 2018 exploring how to combine various technologies to capture, interpret, and fact check live broadcasts from television news stations, using Amazon’s Alexa personal assistant device as a low-friction way to initiate the process. The ultimate goal was to build an Alexa skill that could be its own form of live, automated fact-checking: cross-referencing a statement from a politician or otherwise newsworthy figure against previously fact-checked statements......

    Continue Reading

  • Northwestern is hiring a CS + Journalism professor

    Work with us at the intersection of media, technology and design.

    Are you interested in working with journalism and computer science students to build innovative media tools, products and apps? Would you like to teach the next generation of media innovators? Do you have a track record building technologies for journalists, publishers, storytellers or media consumers? Northwestern University is recruiting for an assistant or associate professor for computer science AND journalism, who will share an appointment in the Medill School of Journalism and the McCormick School...

    Continue Reading

  • Introducing StorylineJS

    Today we're excited to release a new tool for storytellers.

    StorylineJS makes it easy to tell the story behind a dataset, without the need for programming or data visualization expertise. Just upload your data to Google Sheets, add two columns, and fill in the story on the rows you want to highlight. Set a few configuration options and you have an annotated chart, ready to embed on your website. (And did we mention, it looks great on phones?) As with all of our tools, simplicity...

    Continue Reading

  • Join us in October: NU hosts the Computation + Journalism 2017 symposium

    An exciting lineup of researchers, technologists and journalists will convene in October for Computation + Journalism Symposium 2017 at Northwestern University. Register now and book your hotel rooms for the event, which will take place on Friday, Oct. 13, and Saturday, Oct. 14 in Evanston, IL. Hotel room blocks near campus are filling up fast! Speakers will include: Ashwin Ram, who heads research and development for Amazon’s Alexa artificial intelligence (AI) agent, which powers the...

    Continue Reading

Storytelling Tools

We build easy-to-use tools that can help you tell better stories.

View More