User testing: how news designers and developers add context to quantitative data

Last week I wrote about how news organizations use A/B testing to help iterate on design elements such as page layout and headline writing-style in order to increase reader engagement. The technique provides essential information about what a reader is doing, but it does have limitations.

“When you’re only looking at metrics you see the what, but you don’t see the why,” said Steve Mulder, director of user experience and analytics at NPR Digital Services.

A/B tests lack context. In designing media products such as a homepage, story, or app, the motivations for user-behavior are as important as the behavior itself. Digital analytics may show that a reader is spending a lot of time on a story page, but not why. Is the story interesting? Confusing? Hard to navigate?

Observing people interact with a news product can help designers create better experiences for them. Observation can take various forms including eye-tracking and remote testing, but one-on-one observations are particularly exciting for newsrooms because they can be done quickly, inexpensively and they are informative.

Here’s how different news organizations have used one-on-one observations to improve specific products.

Washington Post

In the summer of 2012 The Washington Post launched a significantly updated version of its iPhone app. Before launching the new app, director of mobile Julia Beizer and her team watched readers use it.

It became clear that some of the new features were not intuitive to everyone. One particularly problematic feature was a customizable bookmarks bar.

The Washington Post added tool tips to its app to help readers understand new features.

To help readers figure out these new features, the team added “tool tips” screens (see image above). While Beizer acknowledges that these screens can be annoying and that not everyone might read them, they allowed the Post to keep and highlight a new app feature it was proud of.

Boston Globe mobile app

The Boston Globe saw a similar problem when designing its new app in June 2013, but found a different solution.

In an early beta interface of the mobile app (the screen on the left below) readers could pick which sections and in what order they would appear on their homepage. For example, if a reader only wanted to see arts and lifestyle, she could choose to display only those sections on the homepage and arrange their order (arts or lifestyle stories first).

User testing motivated the decision to eliminate sorting icons from the Boston Globe mobile app.

Testing showed that no one knew what the sorting icon (the horizontal lines stacked on top of each other) meant and that none of the users were re-sorting the sections, according to Damon Kiesow, senior product manager at the Boston Globe. Instead of adding a feature like explanatory text or tool tips screens, the Globe got rid of the problematic feature.

“UX testing through the development process is more about removing features and removing obstacles than adding features,” Kiesow said.

The process was simplified in the next iteration (the screen on the right) to eliminate this problem. Instead of allowing people to sort the homepage sections and the order that the stories appear in, readers could select which sections they wanted and the stories would automatically show up in reverse chronological order.

The homepage of the app went from being an editorially ordered page to a most-recent stream of stories. Kiesow believes that this was preferable for a particular audience accustomed to news feeds and updates from services like Twitter and Facebook, in which users are shown the latest information and not necessarily what they are most interested in.

NPR Apps

Brian Boyer’s team at NPR is working on relatively small projects that take around a week to build. While there is sometimes a rigorous usability test approach, it’s more often “off-the-cuff," he said. This could mean asking people walking by for help or going to someone’s office and having them test a project.

NPR’s Fire Forecast is a map app that “shows current large wildfires and forecast burning conditions in the lower 48 states.”

The NPR News Apps team went through several rounds of prototyping with Fire Forecast.

In early iterations (far left image) buttons were flat and the search box had no “go” button. Boyer and his team tested the app with different people and learned two important lessons.

First, it was evident that people were not interacting with the buttons, because they were not designed to look like actual buttons. In response, the team defaulted toward standard-looking controls that people recognized.

“The second we popped those buttons out, people immediately knew what to do,” Boyer said.

Second, the search box on a phone acts like a submit form in which you can type and then hit search. There was no obvious way to take action without a “go” box, so one was added in. These changes helped readers take full advantage of the app’s features.

What I’ve learned so far

Exposing products to users early on can help a news organization make sure that they are releasing and developing them with audience focus. One-on-one user-testing can help product development teams make informed decisions around what functions or designs to add or eliminate to aid in the larger mission of getting content to reader.

As Alex Wright, former director of user experience and product research at The New York Times explains, “It’s not really about testing the editorial content, but about figuring out the most effective way to deliver that content, and helping users find stories that are interesting to them.”

As evident from the examples above, the formality of the test doesn’t necessarily correlate to how informative it is. This means that whether you can afford to hire a firm to help test products, or if you can only dip into a local coffee shop to run through some task analysis with a willing participant, there’s valuable insight to be gained. After all, the NPR News Apps team didn’t need to hire a company to gain insight into the fact that people were having trouble figuring out how to conduct a search.

Last week I wrote that “design decisions can and should be rooted in data.” The quantitative data obtained from digital analytics works hand-in-hand with the qualitative data gathered through the kind of observational research and one-on-one testing that news organizations are doing. These two strategies for usability testing work together as on-going techniques for improving products.

When designing for the user-experience the why something is happening of qualitative data is just as important as the what that quantitative data shows.

About the author

Jessica Soberman

Graduate Fellow

Latest Posts

  • With the 25th CAR Conference upon us, let’s recall the first oneWhen the Web was young, data journalism pioneers gathered in Raleigh

    For a few days in October 1993, if you were interested in journalism and technology, Raleigh, North Carolina was the place you had to be. The first Computer-Assisted Reporting Conference offered by Investigative Reporters & Editors brought more than 400 journalists to Raleigh for 3½ days of panels, demos and hands-on lessons in how to use computers to find stories in data. That seminal event will be commemorated this week at the 25th CAR Conference, which...

    Continue Reading

  • Prototyping Augmented Reality

    Something that really frustrates me is that, while I’m excited about the potential AR has for storytelling, I don’t feel like I have really great AR experiences that I can point people to. We know that AR is great for taking a selfie with a Pikachu and it’s pretty good at measuring spaces (as long as your room is really well lit and your phone is fully charged) but beyond that, we’re really still figuring...

    Continue Reading

  • Capturing the Soundfield: Recording Ambisonics for VR

    When building experiences in virtual reality we’re confronted with the challenge of mimicking how sounds hit us in the real world from all directions. One useful tool for us to attempt this mimicry is called a soundfield microphone. We tested one of these microphones to explore how audio plays into building immersive experiences for virtual reality. Approaching ambisonics with the soundfield microphone has become popular in development for VR particularly for 360 videos. With it,...

    Continue Reading

  • Prototyping Spatial Audio for Movement Art

    One of Oscillations’ technical goals for this quarter’s Knight Lab Studio class was an exploration of spatial audio. Spatial audio is sound that exists in three dimensions. It is a perfect complement to 360 video, because sound sources can be localized to certain parts of the video. Oscillations is especially interested in using spatial audio to enhance the neuroscientific principles of audiovisual synchrony that they aim to emphasize in their productions. Existing work in spatial......

    Continue Reading

  • Oscillations Audience Engagement Research Findings

    During the Winter 2018 quarter, the Oscillations Knight Lab team was tasked in exploring the question: what constitutes an engaging live movement arts performance for audiences? Oscillations’ Chief Technology Officer, Ilya Fomin, told the team at quarter’s start that the startup aims to create performing arts experiences that are “better than reality.” In response, our team spent the quarter seeking to understand what is reality with qualitative research. Three members of the team interviewed more......

    Continue Reading

  • How to translate live-spoken human words into computer “truth”

    Our Knight Lab team spent three months in Winter 2018 exploring how to combine various technologies to capture, interpret, and fact check live broadcasts from television news stations, using Amazon’s Alexa personal assistant device as a low-friction way to initiate the process. The ultimate goal was to build an Alexa skill that could be its own form of live, automated fact-checking: cross-referencing a statement from a politician or otherwise newsworthy figure against previously fact-checked statements......

    Continue Reading

Storytelling Tools

We build easy-to-use tools that can help you tell better stories.

View More