Scraping web pages is a well documented process. There are plenty of guides on how to pull information using plugins like Python’s Beautiful Soup or browser extensions like Kimono. Many web applications even provide public APIs for gathering information, such as Facebook’s Graph API. Yet, there is a growing set of popular mobile apps that do not have a public API. Apps like Yik Yak, Tinder, and others contain a wealth of information about the...
The Guardian's "The Counted" documents law enforcement killings in the United States. On Nov. 15, Richard Perkins was fatally shot by officers in Oakland, California. His death marked the 1,000th entry in The Guardian’s The Counted’s database that now includes more than 1,063 names. The project launched June 1, and has quickly won acclaim for its relatively robust law enforcement killings database, which is generated via tips submitted on the Guardian website or on social...
Ideas |
While virtual reality has just recently emerged as a storytelling tool for journalists — The Columbia Journalism Review even calls it “journalism's next frontier.” — filmmakers and gaming enthusiasts have been experimenting with the technology for much longer. To get a sense of where the technology might be headed and the challenges journalists are likely to face as they adopt the technology, I talked with two people who have been working with VR for quite...
Ideas , Profiles , Apps & projects |
The website for the US Digital Services Playbook looks like anything but a typical government website. Find it at playbook.cio.gov/. In September, 18F, a team of designers and developers within the General Services Administration (GSA), and the United States Design Services (USDS) released the US Web Design Standards, a project that aims to unite all government websites under a single set of guidelines that guides visual design and user experience. This idea of creating a...
Ideas |
As a typical #millennial I get most of my news from the organizations and friends I follow on Twitter and Facebook. I’ve also started regularly checking Snapchat Discover for curated entertainment content at my fingertips. This year, Facebook, Twitter, Google, Snapchat and Apple all released new news aggregation and presentation features and apps for mobile. There are so many new ways to get your daily dose of news and information that it can be overwhelming to...
When Broadly, Vice’s female-centric vertical debuted on August 3, 2015, I was struck not just by the kinds of content they were putting out, but also by its clean yet personable design that complemented its unique voice. Unlike the heavy black color scheme and font weights of Vice Media’s other sites, Broadly was bold in its use of color, typography and grids. I was curious about how Vice designed and launched Broadly, a site that...
Ideas |
These days curiosity is likely to lead you on a long trek through the depths of the Internet. You read one article and you stop at a shiny hyperlink that screams, “click me!” Before you know it, you are 30 tabs deep and way off topic. I value these journeys for the unexpected treasures that lie along the way, but sometimes the connection of that treasure to your origin isn’t clear. Though you have the...
I was an avid gamer when I had the luxury of time, which is not an affordance you can indulge when you go to Northwestern. The games I enjoyed the most were those that not only had me immersed in a new world but the ones that had me learning along the way. I can vividly recall enjoying the Assassin’s Creed series because of its (somewhat accurate) historical context. Even though I was fighting as...
Ask me what my biggest take-away was from hours of logging tape for NPR’s ongoing series on the United Nations Conference on Climate Change in Paris, and I will say, “If you want to ensure that your audiences are educated, empathetic and responsive, responsible you'll have to speak their language." In the past few weeks, I’ve listened to (and rewound, slowed down, and re-listened to) some of the world’s leading scientists describe what went right...
At MozFest you’ll hear the word “open” a lot – open data, open news, open knowledge, open science, open web and so on. I wondered, though, what does “open” really mean? A few years ago, I found out about open source software. My understanding at the time was that open sourcing a project was simply allowing other people to see its code. That simple understanding was enough to get by. But after hearing “open” prepended...
As data journalists, we tend to focus on visualizing our numbers beautifully for desktops. We pour over D3.js line charts and spend hours getting the tooltips on our maps just right. And right before our deadlines, we’ll throw in some CSS media queries for mobile screens and call it a day. I know I’ve been a culprit more than once. One of my favorite sessions was Aaron William’s MozFest session “Crafting new visualization techniques for...
For some reason, in my head, I’ve always had a really vivid image of what a hackathon might look like: a conference for brilliant individuals (who probably self-taught themselves how to code when they were 11-years-old), madly typing away on their computer and seeing possibilities that I couldn’t see. I’d probably meet the next CEO of Silicon Valley’s newest tech start-up. Or, meet other superhero geniuses in the form of young 20-year-old bodies. So, when...
Events |
It’s no secret that newsrooms are increasingly using bots to cut down on busy work. Software now routinely churns out quarterly earnings stories for The Associated Press and earthquake alerts for Los Angeles Times, freeing reporters to pursue more in-depth projects. And while no bot can write 3,000-word investigative stories, it can assist reporters by alerting them to new data and filtering the information for them. While the idea of creating a bot may sound...
Ideas |
I have been a journalist and journalism student for a decade. When I started my first professional job, in 2007, the dream of every young reporter was to have a scoop stamped on the front page of the paper the next morning. At that time, of course, we didn't have the Internet on our phones and social media was 100 percent something called Orkut — at least in Brazil, where I am from. For years I worked reporting, writing,...
Ideas |
A year ago, I was struggling to choose between offers to study journalism or computational analysis and public policy. I ultimately chose computational analysis and public policy as I was excited by the prospect of learning about the frontiers of data and policy. I quickly developed a fascination with civic hacking and the open government movement. Aspiring to use my data science skills for social good and to return eventually to journalism, I applied for...
Knight Lab alumni fellow, Farahnaz Mohammed, received a nice bit of news recently, when she learned that she’d made the cut for the inaugural class of Knight-VICE innovators. Mohammed is among four young journalists selected to develop innovative ideas in journalism that’ll be supported by VICE, Knight Foundation, and City University of New York. Farahnaz Mohammed Mohammed, who graduated from Northwestern’s master’s in journalism program earlier this year, will continue work on a platform she...
Through a unique scholarship program, the Medill School at Northwestern University and the John S. and James L. Knight Foundation have brought a dozen talented developers into journalism over the last several years. Successful as these scholarships have been, the program could benefit only those technologists who were able to take a year out of their lives to study journalism in Medill's full-time master's degree program. Now that's changed: Money from the Knight Scholarships is...
Lab projects , Ideas |
Since joining Knight Lab as student fellows in April, Michael Martinez and I have been thinking about podcasting technology and online audio in hopes that a project idea would emerge. We're obviously not alone. The last 12 months have seen the rise of highly popular shows like Serial and advanced mobile or in-car "podcatching" platforms. Just last week, podcasts marked another milestone when President Obama appeared on Marc Maron's WTF podcast. Unfortunately, a handful of...
In April, Washington Post announced that it had set a new single-month traffic record, with more than 52 million unique visitors. The figure represented not only a new record, but also a 65 percent year-over-year gain that led other big-name publishers, according to the Post. Publisher Frederick J. Ryan praised the addition of new editorial staffers and awards, and then called special attention to engagement: While unique visitors were up 65 percent, pageviews were up...
Ideas |
Knight Lab squad at NICAR15. Photo by Anne Li. Until college, I never thought coding was for me. I never intended on learning about the “push” and “pull” of GitHub. I was perfectly content not knowing about the existence (and immense power) of the web inspector. I simply fell into it. It’s been exciting to learn and build new things, but part of “falling into” coding has meant that I can’t seem to shake that...
Tomorrow, Wednesday, June 3, journalism and computer science students in the latest version of Northwestern’s “Collaborative Innovation in Journalism and Technology” class will unveil the prototypes they’ve built over the past 10 weeks. You’re invited to see what they’ve come up with. The projects they will be presenting are: Composite: A provocative project that shows the faces of individuals around a location based on publicly available photos. ImageTweet: A tool that makes tweeting pictures with...
When I first showed up at Knight Lab, I heard a lot about a thing called Bootstrap. In particular, one student fellow was hesitant to use it, much to the disbelief of the staff and guests who dropped by. “How can you not use Bootstrap?” one said. “Practically half the internet is built on it.” Well, I thought. That sounds important. I’d best try this out. While I can’t speak to the statistic of half...
I have a fear of failure. I don’t like it when my inbox has more than five emails sitting in there (at the time of this first blog post draft’s conception, it was at 400). I cringe when my desktop is cluttered with screenshots and downloads. These, for me, are true indicators of the clutter that accumulates in life and how success is measured by organizing that clutter very, very well. Let me be clear,...
Through sheer luck and a generous professor, I nabbed a two-month fellowship at Knight Lab after I graduated from Northwestern’s graduate journalism program, despite considering myself a web development newbie. I knew full well what the Lab did, but I never imagined getting involved on the development end. My aim at the start was to research a potential digital tool to aid in the practice of journalism and hand over a stack of paper to...
Since this post was written, Amazon has launched AWS Certificate Manager, which provides certificates at no cost and substantially simplifies managing them for use in the AWS context. We recommend that readers investigate the AWS Certificate Manager product before following the guidance in this post. Every day, more and more websites are serving their pages using HTTPS. This can lead to warnings or complete failures when those sites want to embed content from other sites. Until recently, that...
Screenshot of NPR's new embeddable audio player. Last month, NPR announced that it would make 800,000 pieces of audio available to embed across the web. While NPR has offered limited embedding since 2009, the depth and breadth of this project is new. It's the first time that NPR will offer a single embeddable player with access to such a large amount of content. The work to design and build the player began earlier this year and was led...
For the second time in three years, a Pulitzer Prize-winning story has featured a piece of Knight Lab technology. The TimelineJS instance the Daily Breeze created to "show the superintendent's machinations through the years." On Monday, the Daily Breeze won the prize in local reporting for its work investigating California’s Centinela Valley Union High School District and its superintendent’s outsize salary. Featured prominently in the series of stories was an instance of TimelineJS, Knight Lab’s...
Ideas , Profiles , Apps & projects |
On Friday, February 21, 2014, two members of a Southern California family dynasty were indicted on a series of political corruption charges, including tax fraud, money laundering, and bribery. Two members of that family — Tom Calderon, a consultant and a former assemblyman, and Ron Calderon, a state senator — would surrender themselves by the following Monday, both pleading not guilty to the charges. Tied to their alleged wrongdoings was an extensive network of people...
On Friday, Knight Lab co-sponsored an event with Kickstarter and Cards Against Humanity, whimsically entitled "How to journalism in a Scary World if you don’t have a Fancy Grant." The event brought together journalists and other storytellers to talk about how they’ve supported their work and the various funding models you might be able to use as well. Nicole He was on hand to make a pitch for Kickstarter, which last year launched a journalism category. It’s obviously...
Ideas |
In the last few months, I cheered from the sidelines of the Macy’s Day Parade, I built a kite to fly in India, I strutted the streets of New York Fashion Week and I tailgated a football game in the dead of January. All from the comfort of my bed. How did I do it? Well, I watched these events unfold via Snapchat, an app I normally reserve for sending selfies to friends. I didn’t see...
Ideas |
This twice-weekly exchange has been going on since my first editor’s meeting two years ago when I first joined the editing team of North by Northwestern, Northwestern’s online student-run news magazine: “We’re going to do a story on the architecture on campus for this Sunday,” a section editor declares excitedly at the editorial meeting. “It’s going to be interactive. It’ll be so cool!” “That’s great,” a managing editor responds. “Have you talked to the interactive...
Profiles , People , Apps & projects |
Occasionally we'll do a Q&A with an impressive maker or strategist from media and its fringes. Each person brings a unique perspective on journalism, publishing and technology. Catch up and/or follow the series here. Up until the end of last year, my daily news experience involved sifting through endless tweets and clickbait on Facebook. When I got an invite to This, a link-sharing social network that aims to “find and share the web’s best stories,...
NICAR |
Machine learning is certainly not a new concept in journalism, but it seemed to enjoy plenty of prominence at NICAR this year — fantastic news for newbies to the field like me. I attended several sessions on it, both theoretical and technical, and a few key concepts came up repeatedly. Whether this year’s conference was your first exposure to machine learning, or you’re a seasoned pro, here are four takeaways worth reviewing: Machine learning is...
If you want to uncover business news, never underestimate the power of looking at datasets that don’t look like traditional business datasets. Speaking in the “Investigating business with data” session at NICAR15, the Wall Street Journal’s Andrea Fuller and CNBC’s John Schoen provided a wealth of examples of stories written using not only traditional business datasets like the SEC's Edgar and FINRA but also non-traditional business data sets such as Medicare.gov. Datasets like Medicare.gov greatly...
Participants in the NICAR conversation, "The 'hybrid reporter' identity crisis?" posted their official job titles and what they want their job titles to be. Despite the fact that neither drinking nor networking (“You need to be more sure of yourself!” Alex Duner wailed to me after an awkward introduction to his former colleagues) is really my thing, I found myself at the bar of the Marriott Marquis, in a circle of journalists from across the...
Some of the coolest investigations that have been published over the last year have made use of remotely sensed data from satellites in space. Space is cool. And journalism is cool. So surely the combination of the two must be interesting. And while it didn’t involve firing rockets with journalists on them into orbit, a session at #NICAR15 titled “Space Journalism” introduced some really cool ideas about using satellite data to find and tell stories....
Data. Interactive. D3. X charts and Y graphs that explain Z. I feel that we’ve reached a point where “interactive” has become an empty buzzword in journalism. It’s amazing how quickly interest and adoption of news apps and data visualization has grown in the last few years (just look at the sheer size of this year’s NICAR conference), but as interactive and data-driven journalism becomes more pervasive, we also need to reexamine the meaning of...
We know data tells us a lot. We write programs to automate data scraping. We spend hours creating data visualizations that help readers see what they need to see. We use data to make claims and generate stories that are reliable and have impact. Data is important and we seem to be surrounded by it. But that's not quite true. Sometimes, there is no data? A session at NICAR that really resonated with me was Data from Scratch:...
NICAR |
A big mistake when dealing with data can ruin your day. Luckily there are simple ways to avoid big mistakes and maintain credibility with your colleagues and your audience. At NICAR 2015, a panel of data journalists from The New York Times, Wall Street Journal and Atlanta Journal-Constitution discussed the road blocks they've encountered when working with data for a story. The panel, moderated by MedPage Today reporter Coulter Jones, featured advice and cautions to keep in mind when dealing with numbers,...
Before I become a good data journalist, even an adept data journalist, there is a skill I really need to nail down: balancing data and anecdotes in an easy-to-understand narrative. Super simple stuff, right? Wrong. It’s a skill that I was drilled and tested on in a handful of courses at Medill, but I don’t feel I've reached the sweet-spot where numbers are present, yet not boring facts that most readers skip. Coming into NICAR...
Knight Lab brought a big crew to Atlanta and we’d love to meet as many of you as possible. If you’re interested in talking some shop, sharing ideas, or just saying hello, keep your eye out for: Heather Billings (presenting PyCar, an introduction to Python, from 9 a.m. to 5:30 p.m. Thursday) Joe Germuska Ryan Graff Jordan Young Mallory Busch Alex Duner Anne Li Anushka Patil Luke Rague Kulwant Saluja Ashley Wu Nicole Zhu (facilitating...
Ideas |
Back in September I finally started to build my portfolio site from scratch. This was back when the extent of my coding knowledge pretty much ended at how to save a document as an HTML page. Making my own website? Forget it; I had no idea where or how to start. Former student fellow KK Rebecca Lai suggested I find a bunch of websites, study the elements I liked through the console (the box that...
News veteran and technology expert Joe Germuska will take on an expanded role at the Knight Lab as interim director. Germuska has served as director of software engineering for the Lab. He previously co-founded OpenGovChicago and was on the Chicago Tribune News Apps team. Knight Lab executive director Miranda Mulligan will take a job in late January as creative director of National Geographic Digital. Mulligan has led the Lab for two years. “Our faculty, students,...