Knight Lab officially took the wraps off of SoundCite this week, our most recent tool for content creators. In a nutshell, SoundCite makes it incredibly easy for web publishers and writers to include in-line audio in their stories.

We released an alpha version a month or two back and WBEZ’s Jim DeRogatis used it to give a profile of Chance the Rapper more depth by allowing readers to hear the lyrics DeRogatis cited in his piece.

Indeed, music reviews were the original inspiration for SoundCite, but we can imagine it as an effective way to  give readers quick access to clips of 911 calls, speeches, or even ambient sound. The only requirement is that the audio be hosted on SoundCloud. Unfortunately, because of SoundCloud’s embed technology, it doesn’t yet work on iPhones and other iOS-enabled devices. But don’t fret! Your text will still appear and the reader will not even be aware of the missing clip.

SoundCite is open-source and available on GitHub. We encourage you to contribute in any way you wish — fork it, send us pull requests or just let us know what you think. Of course, there’s nothing wrong with just using the tool, either.

And if you find it useful, send a thank-you to SoundCite’s creators: Knight Lab student fellow and Medill student Tyler Fisher and Medill assistant professor Jeremy Gilbert.

Check out the video above for a quick tutorial on how SoundCite works and be sure to send us any feedback: knightlab@northwestern.edu.

Slimformation: A prototype that tracks the kinds of content a user is viewing and provides advice on how to improve his or her “information diet.” Activities tab shown above.

How many of you have tried to diet before?

I know I have my fair share of attempts.

So we all know there are better and worse foods for you (say, vegetables over macarons). The same logic applies to information. We live in a world of information overload, and consuming more of this information doesn’t necessarily make us smarter. In fact, it can do the opposite.

People need to be more conscious about the information they’re consuming. Because it’s really easy to go through life passively consuming information without really getting any smarter. In order to really hone and develop critical thinking skills, we all need to be aware of the kinds of information we’re putting into our body.

As it happens, we built an app for that.

This past quarter, I took a class titled “Collaborative Innovation and Journalism in Technology,” a practicum-type multidisciplinary class that puts programmers and journalists on projects for the duration of the quarter. It’s co-taught by two Knight Lab faculty members, Larry Birnbaum and Rich Gordon.

My team (Basil Huang, Gursimran Singh and Amelia Kaufman) was assigned a project based upon an idea originally sketched by a team that included the Lab’s Joe Germuska and Larry Birnbaum at last year’s MozFest election-hacking session, which was inspired on Clay Johnson’s book, “The Information Diet: A Case for Conscious Consumption.”

“The world of food consumption and the world of information consumption aren’t that far apart: Both the fields of cognitive psychology and neuroscience show us that information can have physiological effects on our bodies, as well as fairly severe and uncontrollable consequences on our decision-making capability.”

The problem of information intake is a really interesting one. There are many things to measure when it comes to reading online, and how to utilize these metrics to quantify engagement.

In brainstorming the interactions of our system, we kept circling back to some main thoughts.

First, diversity of news sources. Sure, you read entertainment and tech and business and politics, but are you always going to the same sources for your news in these categories?

Second, reading level. Hopefully it’s above that of a fifth-grader. Are you only reading short tweets? Two hundred-word blurbs from Buzzfeed?

Third, news categories. This one was pretty obvious, but providing a breakdown for users of the topical categories for the content they’re reading online.

We also wanted to provide Mint-like functionality to allow users to customize these measures and set goals for themselves. Let’s say I don’t really care about sports, but I don’t want that preference to negatively count against me.

Another important feature we decided on was prescription. Sure, we can display a bunch of pretty charts to help visually quantify a user’s information intake, and aggregate the top news sources per category. But in order to provide more utility for our users, we wanted to build some sort of recommendation system, one that takes into account both a user’s reading habits (based on what we’re tracking), as well as the user’s predefined goals.

We built a Chrome extension to help users read better and smarter.

It does three things, primarily:

1. It automatically tracks a user’s reading activity, visually displaying this information and calculating the top site of news consumption per category.
2. It allows users to set goals for their reading activity to help them change their habits.
3. It recommends action items for users to change their reading habits based on their predefined goals and reading activity.

Goals

Prescription

After installing the extension, it runs run in the background and monitors the user’s browsing activity.

So let’s say I navigate to a Buzzfeed article. Our system extracts the page content and URL, runs this through a categorizer and calculates the reading score of the article. We then store this page information and analysis in the browser’s local storage (meaning it’s all on the client – so it’s fast and private). I can then interact with this data in the extension’s popup, where we display a visual analysis of the numbers and provide direction and recommendations for how to improve.

The alpha version is available here. Try it out! We’d love to hear your thoughts and feedback, which you can send here: knightlab@northwestern.edu.

Wireframing activity at #confabMN

Journalism has some of the same problems, just backwards.

There’s plenty of content, but it needs to be valuable to readers and monetizable for business purposes. Maybe, if a content strategist peeked at the problems newsrooms are facing, or if our newsrooms editors were better content strategists, she/he might conclude something like the following:

• Journalists should conduct user testing and audience research to improve storytelling experiences for future iterations.

Too often, journalism projects have two problems: (1) More complicated pieces of content are treated like one-off projects. It would take a little extra time initially, but research and user testing after launch may actually save time and improve future iterations. And (2), content is often created for the editor, not the reader, under the (perhaps mistaken) assumption that the editor has fantastic news judgment. Often this knowledge seems to be achieved with surprisingly little feedback, interviewing or user-testing. After all, readers aren’t just readers anymore: they’re users now, too.

• Newsrooms should plan for interdisciplinary involvement earlier in the process to allow for pieces that realize fuller potential.

This is a great idea — it’s just super difficult to pull off in reality. And it’s even more difficult to do it more than occasionally. But we should keep trying.

• Banner ads suck.

In fact, you’re about 478 times more likely to survive a plane crash than purposefully click on one (Thanks, Jared Spool). But when you (the user) don’t pay for a product, you become the product; pay for content or your eyes will be sold. That’s why there are metered paywalls and non-profit alternatives that try to avoid the ad model. But Spool also pointed out that content itself can serve strategic business priorities. Sites should be designed so as to display navigable, sharable content that sells itself, reaches more people on its own, and maintains its value. How to do that? Realize that to design content is to design a service, an experience for your users.

• Zoom out.

Content strategy says that we need to zoom out and think more at the beginning of the process. We need interdisciplinary involvement (writers, developers and designers working together from the start) and user-based decision-making (considering real-life use cases and multiplatform presentation). We need to keep macro goals in mind, even when all-consuming projects lend themselves too easily to tunnel vision.

I’ve learned some of these skills applied to journalism in a couple of my classes in Medill (shout out to j-gilbz and his focus on human-centered storytelling and media product design). And there are plenty of amazing news teams doing these things — producing amazing work that speaks to the future and takes its users into account. And we still have plenty of major problems: the experience of commenting sucks, only the New York Times can pull off metered paywalls, etc. But taking a step back and finding a new strategy from the beginning might be a wonderful place to start solving them.

The problem is that it’s difficult to get all that talent collaborating and working on the right problems. That’s why Ryan Briones, who does civic development for the City of Chicago, came up with a new idea he calls the Civic Needs App. With the help of Scott Robbin and Chris Gansen, Briones aims to build an app that connects regular people who have ideas for improving their communities with developers who have the time and skills to build out those ideas.

Briones understands how hard it is for a developer to identify the right civic problems because he’s been there himself. He was working at a consulting company when he met Paul Baker and Derek Eder and built his first civic app, Chicago Lobbyists, “an open data, open government, and open source” project that helps users see connections between the City of Chicago and lobbyists and their clients.

From there, he became very interested in civic development, but wasn’t sure what to do next.

“There’s not always a way to take the excitement and turn it into something useful or something that’s going to keep you interested in doing a project,” Briones said. That interest is key — these apps ideally include problems that are useful for the community, but also problems that developers find interesting to solve.

Foodborne Chicago, for example, is one of Briones favorite civic apps. It encourages people to file food poisoning reports, and, from a technical perspective, it’s interesting because it proactively finds people who might have food poisoning through their tweets and a bit of machine learning.

Now, because he’s working for the City of Chicago, Briones sees opportunities everywhere. “I’m more flooded with things to do than I am searching for questions to answer,” he said.

That’s where the Civic Needs App comes in.

Briones and his team have created a simple form for submitting a civic need and is now working through a “best workflow,” or the best way to go from someone proposing an idea, to someone building the idea to, finally, someone seeing a display or library of successfully completed projects.

The proposed solution relies heavily on Github, partly because Briones, Robbin and Gansen don’t want to rebuild something that already exists. Contributors will need a Github account, all data will be stored on the site, and when someone raises a civic need, for example, it’ll be stored as a Github issue.

The next step in the project’s development is a more friendly interaction layer and perhaps even a project template: a foundation of code upon which different civic apps can be built.

Collaboration and sharing is standard practice in the civic development community (look, for example, at Chicago’s Adopt a Sidewalk and Boston’s Adopt a Hydrant, which share code), but with the Civic Needs App there will be an easy, standardized way to share code.

That means the Civic Needs App is as much about technology as it is about community. “We want to demystify the process of building an app,” Briones said, “and get people who have ideas but who aren’t necessarily developers a little more involved in the development process.”

This sounds surprisingly similar to the journalistic goal of keeping regular citizens informed and involved. While journalism has historically articulated a community’s issues while civic development has provided practical solutions, “the line between them is blurring,” Robbin said.

“Journalism’s tool belt now includes software that culls facts from massive data sets,” he said, “ and civic developers realize that their software needs to be put in context of a compelling story to be useful.”

Journalists, developers and other people are all trying to solve problems and improve their communities, and the Civic Needs App is one innovative way to do just that.

To learn more or get involved, watch Briones present the Civic Needs App at 1871 in the video, or check out the app’s repo.

The famous GitHub logo.

If you’ve been hanging around newsrooms or journalism classrooms lately, you’ve probably heard the word GitHub. It might sound a little scary and mysterious, but even the most traditional pen-and-notebook journalists should know about this super helpful tool (to say nothing of aspiring newsroom programmers).

So, what, exactly, is GitHub? Why do you need to get it?

GitHub is a social coding site. Designed for the purpose of democratic and collaborative coding, GitHub allows multiple people to work on the same code at the same time.

You can host your own code (called a repository), fork others’ code and suggest changes (more on that below), and accept changes and merges into your own code.

Have you ever seen Octocat (GitHub’s unique cat/octopus mascot) badge on someone’s portfolio? That’s GitHub. Clicking on that will lead you to someone’s GitHub profile, where you can explore all of their publicly hosted code and as well as other projects they’ve played around with.

But beginners should take note: GitHub isn’t Git. And knowing the difference is crucial to developing a more productive workflow. Git is revision control software, originally developed by the Linux community. GitHub is a web-based hosting service that takes Git’s software and pairs it with an easy-to-use interface that requires almost no previous coding experience to use.

Get familiar with code

GitHub is an ideal platform for beginning coders because the site is easy to use and version control can act as a safety net as you try new new things. And if you can’t quite figure out where you’ve gone wrong, the site provides users with plenty of avenues for easy troubleshooting on new projects.

David Eads of the Chicago Tribune Newsapps team says you can think of GitHub like a filing cabinet. After a project is published, it’s easy for others to go back and examine the code, learn from it and make it better. Moreover, GitHub’s Gist feature is a quick way to jot down bits of code you want to remember or show to someone for another opinion.

And don’t forget, GitHub is called a social network for a reason. Newcomers can watch code as projects take shape or troubleshoot with more experienced users. Groups like OpenHatch aim to answer coding questions and maintain a set of projects on GitHub that are helpful for beginning programmers.

Beyond GitHub’s operational utility, it serves as yet another way journalists and programmers can work together and create innovative projects.

“In a lot of cases, journalists and programmers need to figure out how to hang out in each other’s worlds better,” Eads said. As more journalists embrace GitHub as a way to improve stories, they’ll develop a new kind of news community, centered around collaboration and code – truly a news nerd’s nirvana.

Some vocab

Before diving into GitHub, get a leg up by understanding its litany of technical terms. You’ll also want to download Git and the GitHub native app for either Mac or Windows. GitHub explains every step on their website.

• Commits — This is simply a change you make to a piece of code, or a set of related changes made on multiple files.
• Forking — Forking is simply the copying of an existing code repository at a specific point so you can add to it as you see fit. The original code still exists for the creator, but your fork may spin off into a lively new project. This means that any project publicly hosted in GitHub can be edited by anyone. For example, if you find a typo in the ProPub Nerd Guide you can change it.
• Pull requests — Pull requests let users know what changes you may have pushed to a particular repository. Once you send a pull request, other users can review your changes and incorporate them into their repositories.

Journalism projects and resources to get you started

Getting on GitHub will open a world of new projects and applications you can repurpose for your own reporting. “Anything cool you can find on the internet you can probably also find on GitHub,” said Zach Wise, a faculty collaborator at the Lab. Adds Knight Lab student fellow Tyler Fisher added, “If an open-source project isn’t on GitHub, there’s a bit of a concern at this point.”

To get started look at these projects and resources for the burgeoning code curious journalist. Several news apps team filled with coding all-stars have compiled guides that define their particular style and manifesto when it comes to creating interactive projects. There are also a bevy of cool projects that can help you trick out your slideshow style, build a better website or flesh out your coding skills.

• NPR Apps Guide
• ProPublica Nerd Guides
• The Chicago Tribune TribApps Guide
• Twitter Bootstrap — A front-end framework from the people at Twitter. Bootstrap makes it easier to design a sleek and responsive website without the busywork.
• Javascript for Cats — Learn Javascript (even if you are not actually a cat).
• reveal.js — A framework to create your own Javascript-based presentation so you don’t need to mess with clunky Powerpoint to make your point.

So go on, get GitHub and get started.

Meeting the lovely Tasneem Raja was the highlight of my week during 2012′s South by Southwest Interactive Conference. Formerly a staff writer at The Chicago Reader and the news apps editor at The Bay Citizen, she is now an editor at Mother Jones leading their interactive storytelling team. She is kind, tremendously witty and scary smart. In fact, Tasneem is perhaps best known outside of the news nerd circle for brilliantly writing about the “Brogrammer” problem and sexism in Silicon Valley and about being a woman in the video game industry last year. Read on to learn about her adventures in the MoJo newsroom, how she manages an interactive news team and her approach to knowledge-sharing between teams of journalists.

Tasneem Raja

Miranda: What are you working on these days?

Tasneem: Our data visualizations and interactive graphics need to look awesome on mobile, and they don’t right now. MoJo’s numbers are the same as everyone else’s: mobile traffic growth is outpacing desktop audience growth. And even if it weren’t, designing interactive stories with mobile in mind is simply the right thing to do. As Brian Boyer has put it, if it doesn’t work on mobile, it doesn’t work. We’re working on an open-source charting template that handles data comparisons in a slick, engaging way (aka D3), and we haven’t done the best job ever of prioritizing a fluid layout. So, it’s time to course correct.

What do you like about Mother Jones?

Tasneem: It’s funny: my first job in data journalism was as a news apps editor at an ambitious startup journalism outlet that garnered a lot of “future of journalism”-buzz when it launched in 2010 (RIP The Bay Citizen). Here I am working at Mother Jones, which I first learned of in high school while visiting a beloved teacher’s house for tea, where I had to step over dusty stacks of old Mother Jones issues to cross the living room. And yet this 36-year-old news organization has this incredibly nimble, risk-loving, and straight-up fun spirit that I’ve found lacking in many upstart media ventures, which supposedly got to sidestep all the legacy problems of old newsrooms.

For instance, I don’t have to convince anyone here that iterative product development is awesome. Just yesterday, I heard a news editor encouraging a reporter to “think iteratively” about how to chase down a major story.

There’s a top-down culture of testing out emerging possibilities and new ways of doing business if there’s a reasonable chance they could further the mission. And, more importantly, when something isn’t working, even if we’ve made a significant investment in the endeavor, it feels totally safe to say so.

Tell us about your team. How is it set up?

Tasneem: There’s three of us on the Interactives team at MoJo. My producer Jaeah Lee is completely amazing. She comes from a reporting background with superpowers in research and policy analysis, and over the year and change that she’s been on my team, she’s morphed into this scarily competent front-end developer, interaction designer, and data wrangler.

She started out at MoJo as an intern on the magazine’s research desk, and I would be completely lost without her deep knowledge and appreciation for how our newsroom actually *works,* from fact-checking to magazine production to the evolution of our web operation. That’s one of my biggest points of pride: our team has worked really hard to insert ourselves into the storytelling process here at MoJo, and my editors-in-chief tell me that we’re totally fused with the editorial DNA of the place at this point. Jaeah’s been really vital to that win, always making sure we’re looping in the fact-checkers, copyeditors, and story editors when we need them and at the right time.

In the news technology space, badass design and development skills get a lot of attention, as well they should. Jaeah can grok whatever new javascript hottness comes along, but a research-driven front-end developer with data analysis skills who’s as comfortable talking to developers as she is with senior editors? All I have to say is: hands off, y’all.

We get to work with delightfully irascible MoJo developer Ben Breedlove, who splits his time between my team and our core technology team, which works on building out the CMS and making the whole site better. (You know how the site never crashed under the traffic tsunami that was our 47 percent scoop? That was all them.) Ben is a veritable machine of open-source news tools who’s coded over a dozen critically useful pieces of software for us (and everyone else!).

My favorite Ben Breedlove production is our javascript-based Choose Your Own Adventure app, which our reporters have used to explain everything from how Citizens United works to the immigration maze. Greatest hits you’ll find at our Github page also include a quiz generator, a utility for pumping Google spreadsheet data into slick D3 chart templates, and “smoosher,” a Grunt task that makes the horrible process of smooshing our carefully crafted, oh-so-delicate project code into our unforgiving CMS templates, which, before “smoosher” came along, resulted in a bloody mess every time. Ben is endlessly curious about how newsgathering works and intent on sniffing out every pain point along the way, and I value his deep empathy for MoJo’s reporters and editors (our “clients,” in a way) as much as I do his scary-fast javascript chops.

Our ethos is “storytelling by any means necessary.” Sometimes that means building reusable tools to answer newsroom needs for which we can’t find a good existing solution. (We are highly allergic to “not built here” syndrome here—if someone else has already built it better, we rejoice and head to the bar.) Sometimes it means spending a couple weeks designing a D3-powered interactive “universe” of campaign finance data, because we think that’s a better way to tell the super-PAC story than plopping a bunch of tables on the page. And more and more lately, it means training our fellow journalists in the same tools we use, from spreadsheets to JSON to GIS. To me, that’s where the real win happens; making everyone’s journalism better a little bit at a time, across the whole newsroom, not just doing spectacularly better journalism every now and then.

What’s one of your biggest wins?

Tasneem: Last year, I started a skill-sharing series at Mother Jones called “Skillz Thursdays,” a one-hour brown bag session where someone at MoJo teaches everyone else about one thing. It might be “how to make a map,” or “how to understand MoJo’s revenue streams,” or “how to not freak out when getting interviewed on-air by NPR about that awesome story you just did.” Since Skillz is organized by the interactive team, there’s a tech-heavy focus, but IMO our best sessions are when a group of folks in our company who don’t often cross paths get to come together and simply chat, in an unstructured, super low-stakes environment. Skillz Thursdays has become enormously popular among our staff, and I have a requests list for future sessions that’s a mile long. (Our CEO just generously offered to supply lunch on the MoJo dime, so I have a feeling they’re about to get even more popular.) We’re currently figuring out ways to archive the sessions, via slidedecks perhaps, and set up a satellite series in our D.C. bureau, which currently videoconferences in.

What’s next?

Tasneem: After the tragic shooting in Aurora, MoJo senior editor Mark Follman wanted to look at data on mass shootings over the last several years to gauge whether this type of incident was on the uptick. Turns out there was nowhere to look; a comprehensive data set on mass shooting incidents in American history didn’t exist. So, we set out to make one. Mark and his team of two junior reporters from the MoJo fellowship program began the exhaustive process of culling very detailed information on these shootings from the last 20 years, from local news reports, calls to local police departments, and research studies on gun violence in America. From make of weapons used, to known history of mental illness among the shooters, to how the weapons were obtained, no one had ever put this information in one place before.

My team worked with Mark’s crew to scrape and patch together various partial data sets on the topic, compile them in one database (we simply used a Google spreadsheet), and analyze the findings for patterns and trends. We also created a map showing where the incidents occurred, with a full case profile on each one. The map and data findings have been cited in countless news reports, including a New York Times op-ed on our country’s escalating gun problem, and was namechecked by Sen. Dianne Feinstein during a hearing on a proposed assault weapons ban.

There’s been a lot of interest in the last few years in journalists scouring publicly available government data sets to monitor our officials and agencies, and a push to force more agencies to put their data online in formats that don’t suck. That’s all awesome, necessary, and yields valuable findings. But we all know that a lot of the stories aren’t just sitting there waiting to be found. I’m personally very interested in the magic that happens when you put unconnected data sets together for the first time, or create new data sets that don’t exist yet.

I’m really interested in the possibilities of “preemptive” data findings: identifying problematic industrial plants via a crosscheck of several regulatory databases before the next explosion, for instance. It’s not quite that simple (and doesn’t really even sound very simple now that I read back), and it’s no new concept—see Philip Meyers and the Detroit Free Press’s survey data collection efforts after the city’s 1967 riots—but it’s one I’m particularly invested in. And, in my opinion, one Mother Jones is in a uniquely good position to mine, given that we still have a hefty fact-checking desk and the groundwork we’ve laid in the last couple years to beef up our data wrangling skills across the newsroom.

What is the biggest luxury in your life?

Tasneem: Last year, for the first time, I moved into my own apartment, a little one-bedroom in Oakland’s Temescal neighborhood with a lemon tree shading the front windows and, gasp! garbage disposal. It’s just me in here, no partner, no siblings, no roommates. This may not seem like such a big deal, but in my family and the culture I come from, it is. Sitting on my couch with a beer and a magazine and quiet all around me feels like an incredible and hard-won luxury.

Follow Tasneem Raja on Twitter, @tasneemraja. Find weekly updates from our profiles series on Fridays.

The North by Northwestern housing guide project we built

Last week, I launched my first team-developed news app with a group of amazing student news nerds and peers: Hilary Fung, Dan Hill, Rebecca Lai, Sheng Wu and Katie Zhu. We developed a housing guide for incoming freshmen to Northwestern University who are in the process of applying for freshman housing. It includes an interactive map of campus, filters to narrow down housing options, photos and videos of each hall, quotes from current residents and basic statistics including number of students, room size, cost, and distance to key campus locations.

The project, annually published by North by Northwestern, has existed in some form or another since 2008 and has been developed by rock stars like Tom Giratikanon (the site’s founder), Sisi Wei and Emily Chow. For the past two years, the project has fallen on my shoulders as the publication’s former interactive editor and current webmaster.

Last year, I foolishly undertook the development process all by myself. The final result was somewhat embarrassing, a barely functional mess of a Canvas-based map with a whole bunch of ugly JavaScript to go with it. It was all I could manage as a much more limited programmer without the access or technical ability to break free of our Django-based CMS.

This year was going to be different. I learned my lesson. I would get help. I would get people better and smarter than me to develop alongside me, and together, we could develop the app I truly wanted to make.

First of all, we needed to present the reporting, photography, and videography of nearly 50 student reporters in the best light possible. They work too hard for the app itself to let them down. After some discussion, the team decided the best solution was a fully independent Django app. Our potential users, incoming freshmen, would use this application from anywhere in the world with any kind of device, so we needed something responsive. We wanted to provide basic information and accurate data on each hall. However, we knew that our advantage over standard university advertisement had always been the opinions of current residents on each dorm and the deep level of access we had to all of the facilities. Thus, we wanted to emphasize the character of every residence hall in this edition of the guide as well.

I think we accomplished all of those things. But it wasn’t easy, even with the ridiculous talent on the team. In my first role as a lead on a team of developers, I learned a lot of lessons. Here are some of them.

Set Early Content Deadlines and Meet Them

One of the best decisions we made early on was to get all content ready weeks in advance. Of course, small changes continued to be made up to the night of publication, but this allowed us to be in a place where we could develop with real data. The inevitably late content still came in at a time that was manageable to integrate in the development process. In a Django environment, we were able to fine-tune our models to our content so that we had nothing extraneous.

This helped our design process too. We knew exactly what we had to work with, so even our initial mockups were fairly accurate. We could iterate and find the best place for each piece of content we had. In a responsive context, we could orchestrate our content in a way that made sense on every device we tested on. On the dorm detail pages, we could play with many iterations of content choreography. In the end, we chose to put the vital statistics (dorm size, location, amenities) in the left rail for a desktop size, but we forced that content below the multimedia elements on mobile. Before we reached that solution, the vital statistics section existed on just about every other area of the page. Knowing what we had early helped us find the proper placement.

Of course, meeting our content deadlines was only possible because of the project’s phenomenal editors: Denise Lu and Megan Thielking.

Use Spreadsheets Because They Rule

To populate our database with our content, our editors filled out a spreadsheet with all of the information we needed, including each individual quotation from a student, an embed code for each video and a link to every photo to go in our slideshows. It was a comprehensible way for them to make sure the right content got into the development team’s hands. And for us, it just took a few django-csvimport commands to get the database running. The smoothness of this process makes me excited to find a project for something like Tarbell.

Pair Programming Is Awesome—Do It Often

Operating on different schedules, we found it hard for all of us to meet in the same place at the same time and hack on the project. This made our code go in many divergent paths before it finally all came back together with about a week until publication. In that last week, I met with just about every member of the team individually for a few hours, and we paired on whatever aspect of the project they were responsible for.

Perhaps it was the threat of deadlines looming over us, but those pairing sessions were ridiculously productive, and I wish I had started having those sessions earlier in the development process. Having me there as the common element between the sessions helped unify the code and make sure everything that needed to get done was completed. I may not have written every line of code, but I saw most of them get written and approved of them.

Ask for Data and Sometimes, Ye Shall Receive

Detail page from the app

As a student journalist, I know covering a private university is hard. Private universities have very little responsibility to disclose what they do, and they generally have no reason to tell student journalists (or any journalists) anything more. With the housing guide, we have often been limited as to what information we could provide because, discouraged from so many other stories, we never asked for the data. But this time, Northwestern’s Residential Services cooperated with us in the interest of providing accurate information on all fronts to incoming freshmen.

Thanks to their data, we could provide accuracy on the number of residents in each hall and what types of rooms are available. We were able to provide clarity on the different types of options available for freshmen. The relationship was mutually beneficial, and as student journalists at private universities, we should look for more relationships like this one where we can practice with data and use it to make something worthwhile.

Assign Roles Based on Expertise, but Get Everyone Involved in Everything

Dan knows his way around a map. Hilary and Rebecca are fantastic designers. Sheng and Katie have a wealth of full-stack development experience. It was easy for me to assign responsibilities to each member of the team based on his or her expertise, and that general structure ensured things got done.

But this experience was meant to teach as well as produce. So when Dan was ready to incorporate his map into the design, we showed him the grid system we used to make the responsive magic happen. When it was time to get Hilary’s design templated into Django, we did it by showing her how the models and views worked so she could begin writing template tags herself. They all came out of this experience with more knowledge than when they agreed to help out over a month ago, or at least I hope so.

Test with Force

The first live day for this app was not great, and most of the blame for the problems lay with me. I made a few dumb mistakes in the Django app that caused the entire server, not just the guide, to crash. The dumbest one was this: to make the map, I decided it was a good idea (despite protest from smarter members of the team) to run a Django view through a jQuery AJAX request that turned the GeoDjango Multipolygons into GeoJSON data that Leaflet could parse. This made our Varnish cache useless, forcing a request to the application on every single page view. Naturally, the server crashed four times in two hours.

Thanks to a massive assist from Jeremy Bowers, I baked out the JSON to static files on S3 and used those to make the AJAX request. I could have avoided this whole problem if I load tested the application with any bit of force. I figured we wouldn’t get enough traffic for the Django requests to matter, but even a few thousand page views crashed our server. I would have known that if I did some load testing before deployment.

Have Individual Humility and Group Hubris

This is the best project I have ever been a part of, leader or not, and that is because I worked with a team of my most talented friends together to work towards an achievable goal. I didn’t try to do this all by myself. Instead, we had a group-wide hubris that pushed our boundaries as developers and designers. As a team, we shot for a professional-grade news application, and, in my ever-so-biased opinion, we succeeded.

The students have been working since April, when I and my Knight Lab colleague, Associate Prof. Larry Birnbaum of the computer science department in the McCormick School formed eight interdisciplinary teams out of the 27 students enrolled in the class (11 journalism master’s students, 13 computer science undergraduates, two engineering master’s students and one undergraduate who’s double majoring in journalism and computer science). We gave them a list of broadly defined project ideas, asked them for their preferences and tried to assign students to ideas they were most interested in.

The projects they will be presenting are:

• ChatterTrack: Analyzes and visualizes what followers of a Twitter account are tweeting about.
• Slimformation: Tracks the kinds of content a user is viewing and provides advice on how to improve his or her “information diet.”
• MusicRx: Modeled after the BookRx project developed at the Knight Lab, recommends music based on the content of a user’s tweets.
• Sensus: Helps journalists find newsworthy data in the U.S. Census.
• twXplorer: Enables journalists to explore and save tweets about a topic they are interested in.
• Timeoutline: Lets publishers tie together multiple articles about a topic over time into a timeline-based navigation system.
• SportsTweet: Intended for a sports-centric broadcast, such as ESPN SportsCenter, visualizes the hot topics being discussed by sports fans on Twitter.
• onMessage: Helps journalists or people interested in politics track the topics candidates talk about on the campaign trail.

In general, we strive for a mix of projects relevant to journalism and media: tools for journalists, software for publishers, and applications that could be useful or fun for media consumers.  We use an agile-development approach; teams are expected to present an updated version of their project each week.  Every team has a functioning prototype now. Between now and the final presentation, the students will be improving and extending what they’ve built so far.

This is the second time Larry and I have taught this course together. It’s a lot of fun for us because we enjoy seeing the journalism and computer science students learn to communicate and collaborate. It’s also fun because many projects end up evolving in ways the faculty could not have anticipated. Some projects end up being further developed by the professional staff here at the Knight Lab.

Please RSVP for the event on MeetUp (Details: 6:30pm (Chicago time), Wednesday, June 12 at the McCormick Tribune Center Forum). If you can’t attend in person, we’ll be streaming the presentation at http://bit.ly/Collab-Inno-Spring13.

We hope you’ll join us, in person or on the live stream. Want to get a sense of the issues the students have been wrestling with? Check out the class blog, Tech Media Street.

I first met Mohamed Haddad two years ago at a media conference in Doha, the purpose of which was to discuss the ways in which technology was affecting media. At the time, few people had a better view of those changes than Haddad. The “Arab Spring” was in full bloom and Haddad’s employer, Al Jazeera, was at the center of it, covering revolutions relentlessly as country after country faced demonstrations — Tunisia, Egypt, Yemen, Bahrain, Syria, and more. With few reporters on the ground in some of these countries, networks often relied on new technology — specifically social media — to get first-person accounts. Here, Haddad talks about working at Al Jazeera during the revolutions and his transition from general technologist to editorial technologist.

Mohammed Haddad

Ryan: Tell us a little about yourself, your role at Al Jazeera and your experience prior to jointing that team.

Mohammed: I’m Palestinian South African and have been with Al Jazeera for a little over two years now. I work as an interactive producer for the website focusing on data driven storytelling. My background is in computer science and information systems. Prior to joining Al Jazeera I was involved in crowdsourcing research and application in Africa.

You work in a part of the world that’s seen many historic news events and big changes in media recently, what’s that experience been like?

Mohammed: I was very fortunate to join Al Jazeera in 2011 just at the time of Egypt’s revolution. The energy was tremendous. I think we all had a sense of how big of a moment in history that was going to be. For me, being fresh in the news business, it didn’t take long to see how the combination of traditional media and technology was able to impact hundreds of millions of people in the region.

What has changed since you started working?

Mohammed: At the time, technology disruption to the media space meant that new newsroom tools had to be developed almost overnight to keep up with real time services such as Twitter. The concept of live blogging was relatively new and was something that Al Jazeera strongly embraced. Now, there is a good range of tools available to journalists to take advantage of live social media conversations. This means that news organizations can be a bit more proactive in how they approach reporting.

Why is technology important to the media industry?

Mohammed: Communication is at the very heart of media. Technology provides the means to communicate. Everyone should read Nick Diakopolous’s “Cultivating the landscape of innovation in computational journalism”

Who is the most creative technologist out there right now?

Mohammed: I’m a fan of the work of Simon Rogers, former Guardian Datablog editor and newly minted data editor at Twitter. There are many exceptional technologists out there, but I think being an editorial technologist is a very powerful combination.

What companies or media outlets that we might not know about are doing exciting, innovative media, tech work?

Mohammed: Locally, the Qatar Computing Research Institute (QCRI) is doing a lot of work in media research and innovation. They have quite an extensive innovation branch which looks at a variety of fields including Arabic language technologies, data analytics, and social computing. Al Jazeera has worked with them on several projects to develop tools to enhance our journalism processes. The Institute’s Carlos Castillo keeps a great blog that looks at applied social computing in the news context.

How can people use technology to create change in the world?

Mohammed: For most people the responsibility lies in sharing truthful messages using technology. For developers there are always opportunities to improve upon existing tools to better pass on that message.

If you could design a technology that would solve any problem in the world, what would it be?

Mohammed: Technology alone doesn’t solve problems, but I’d like to design a platform to better manage crowd-sourced information.

What is the biggest tech challenge that media companies will face over the next 5 years?

Mohammed: Monetization is one of the biggest challenges for many news organizations and doing that will depend in part on another big challenge — measuring impact and reach on a very decentralized web. Apart from that, keeping up with technology platforms, specifically how journalism workflows are impacted is something the industry ought to consider as big data finds its way into the newsroom. And finally, social media verification and data interpretation is something that many media companies will have to embrace.

What do you read/watch/consume religiously?

Mohammed: I’m a regular attendee of O’Reilly Webcasts. I’ve got quite a good collection of Javascript books and bookmarks, which I enjoy learning from for new projects. Also, I’m quite a fan of Google Fusion Tables that I’ve used since day one.

What applications do you have open while you’re working?

Mohammed: I have a 27-inch iMac at work. On the top left of the screen I have Twitter and Chartbeat pinned as tabs. Google analytics is just a bookmark away. For projects, I use TextWrangler or Sublime Text, Photoshop or Illustrator, and usually Chrome and Firefox with a couple dozen tabs open.

What could the world use a little more of?

Mohammed: Patience.

What could the world use a little less of?

Mohammed: Impatience.

The lightning talks at NICAR are often the highlight of the computer-assisted reporting conference, but Chase Davis (who recently did a Q&A with us) really grabbed my attention with his “Five Algorithms in Five Minutes” talk, complete with a mic drop. So much so, that three months later I’m still thinking about it and all of the ways that I might put these algorithms to use.

NICAR coincided with my internship at The Sacramento Bee, my hometown paper, where I was spending lots of time with–and eating much Chipotle with–computer-assisted reporter Phillip Reese. Reese has become the go-to data expert in the Bee newsroom by helping reporters find numbers to back up their stories. He knows how to not screw up data and bulletproof his spreadsheets by keeping track of your records, making backups and asking the experts. I admire his ability to find newsworthy trends and outliers with averages, medians, percent change and sorting spreadsheets such that I share his articles on Twitter with #PhillipReeseFanClub.

So when Davis showed NICAR how algorithms can help reporters dig through data, I thought of our two-person data team in Sacramento. My machine learning research at the Knight Lab and study of algorithms in computer science classes have further shown me how we could have applied these data techniques at the Bee.

Reese and I worked on several projects involving data for all 58 counties in California, so when the state finance department dropped a report predicting how each county’s population will change by 2050, we dove in to analyze (and map) their findings for the next day’s paper. We looked at percent change and tried to find interesting outliers, but that meant keeping track of dozens of demographics variables.

Principal Component Analysis (PCA) could have really helped us out that day. It compresses a correlated variables in a dataset to make interesting variables stand out. In that story we found that the state predicted almost every county to grow due to growths in the Hispanic population, PCA would have singled out those counties that bucked the trend.

Reese localized the gun violence discussion in a Sunday A1 story by profiling Lemon Hill, the Sacramento neighborhood with the most reported assaults with a firearm and shooting into a building. Among the ornery comments on the Sacramento Bee website, some noted that police report data could be influenced by variables like population size, because it’s logical that more shootings occur where more people live. Although Lemon Hill leads all neighborhoods in gun crimes, Multidimensional Scaling, similar to PCA, could certify for its rank as a dangerous neighborhood by controlling for factors like population.

When rumors began at the start of my internship that Sacramento’s professional basketball team would move to Seattle, Reese found population data to compare Sacramento to other NBA cities. An implementation of the nearest neighbors algorithm could create a similar comparison, but using more variables — like income or geographic size — to find cities that are comparable in more ways than population size.

I spent hours of my internship building a hexagon map of homicides and shootings in Sacramento. Hexagonal binning is a popular mapping technique because they generate more clusters points, isn’t too difficult to render on browsers, and just plain looks cool. But I also could have used a dbscan algorithm to show concentrations of shootings. Davis’s Python script takes latitude and longitude pairs and creates clusters based on a provided distance.

I also spent hours of my internship examining more than 100,000 PDFs from the state transportation agency for an investigation of construction on the Bay Bridge. PDF scraping technologies like Tabula could have saved me lots of time, but I would have loved to run a Locality Sensitive Hashing script to find similarities in the text.

Computer-assisted reporters know performing operations on data requires clean, structured spreadsheets. These algorithms are no exception. Davis’s scripts for machine learning algorithms all use Python libraries like numpy and scikit-learn that apply algorithms to data in CSV files. Installing the necessary libraries and editing the Python code to run these scripts on your own datasets means flexing some programming muscles, but if computer-assisted reporters can find front-page stories in seas of census data, they can wield a command prompt and take next step in data analysis.

After spreadsheets, algorithms are the logical next step.