MozFest 2013: Journalists should command the command line

Journalists who want to learn more technology often jump into HTML, CSS and Javascript. Those are great places to start (as Knight Lab and others have written before), but if you want to maximize the potential of your computer, one of the first things you should learn is the command line!

Some quick background: Regular computer users access the computer via a graphical user interface (GUI). This interface allows you to interact with the machine using a mouse and images on your screen to make the computer do what you want.

But you can also navigate and control your computer by using text-based commands on something called the command line. By firing up a program called Terminal (at least on Mac), you can enter text commands and navigate through your computer.

Noah Veltman, a 2013 Knight-Mozilla Fellow at BBC News, led a session at MozFest called “Solve A Murder Mystery on the Command Line." The game involved a folder of .txt files with thousands of lines. To solve the murder mystery, we had to search through the files using the command line to pull out specific phrases.

After channeling my inner detective during the session, it occurred to me that I didn’t know how to use the command line until a couple months after I started programming. Once I started using the command line, I understood my machine much more and learned programming more quickly.

I talked to Noah after the session to gather a couple more ideas on why journalists should learn the command line.

It makes it easier to work with text data


When it comes to working with text data, especially large and messy files, command line tools are your best friends. Datasets might come to you as .csv files from a government agency, or as .txt files from local companies. To navigate through these files or to extract useful and interesting information the command line helps you do so quickly.

You might wonder, Why not just open Excel or Access to look at the data? While Excel works great with a 50MB file, the software isn't nearly as nimble with a 5GB file. By way of example, Illinois School Board School Scorecard files include more than 10,000 columns and precious little metadata to explain what’s contained in each one. The big and complicated files are very hard to comprehend and it turns out to be a challenge to analyze the data.

With the command line, a couple basic commands will help you analyze text data:

cat filename
The cat command allows you to read the files. It opens a text file and prints the content in terminal. Type the word cat followed by a space and the name of the file.

grep option pattern filename
The grep command is used to search text for specific patterns. For example if I would like to search for the word “corgi” in a file called corgi.txt. I would enter the command:
grep “corgi” corgi.txt.

head option filename
The head command reads the first few lines of any files. For example if I would like to look at the first 20 lines of a file I would enter the command:
head -n 20 corgi.txt

It lets you use tools built just for journalists


Tons of tools built for journalists involve using the command line, if only during the installation phase. You’ll often hear that “software package X is exactly what you need" for a particular task, but then you wind up looking at a GitHub repo where you learn that installation of that software involves various command line operations.

csvkit, for example, converts files to .csv and helps you to clean up and standardize data. Using csvkit and the command line, you can filter a .csv down to a subset of columns, search and filter rows, join various .csv files, etc. These tasks are done by simple commands like in2csv, csvcut csvgrep, etc. As opposed to copying and pasting from one Excel file to another, these short commands allow you to clean and organize your data in a matter of seconds.

It’s a gateway to full scripting languages


Noah referred to command line tools as “a gateway drug” to actually learning a full scripting language like Python. Once you begin to feel comfortable with a few basic commands, you can begin to combine them in all sorts of ways to get what you need even if you cannot write a custom data processing script.

There are plenty of reasons to learn the command line. Hopefully your curiosity is sparked. If it is, click the links below to learn more:

http://cli.learncodethehardway.org/book/
http://www.youtube.com/watch?v=Fzn6jbaw6O0&feature=related
http://www.linuxjournal.com/content/downloading-entire-web-site-wget
http://csvkit.readthedocs.org/en/latest/

About the author

KK Rebecca Lai

Undergraduate Fellow

Latest Posts

  • A Big Change That Will Probably Affect Your Storymaps

    A big change is coming to StoryMapJS, and it will affect many, if not most existing storymaps. When making a storymap, one way to set a style and tone for your project is to set the "map type," also known as the "basemap." When we launched StoryMapJS, it included options for a few basemaps created by Stamen Design. These included the "watercolor" style, as well as the default style for new storymaps, "Toner Lite." Stamen...

    Continue Reading

  • Introducing AmyJo Brown, Knight Lab Professional Fellow

    AmyJo Brown, a veteran journalist passionate about supporting and reshaping local political journalism and who it engages, has joined the Knight Lab as a 2022-2023 professional fellow. Her focus is on building The Public Ledger, a data tool structured from local campaign finance data that is designed to track connections and make local political relationships – and their influence – more visible. “Campaign finance data has more stories to tell – if we follow the...

    Continue Reading

  • Interactive Entertainment: How UX Design Shapes Streaming Platforms

    As streaming develops into the latest age of entertainment, how are interfaces and layouts being designed to prioritize user experience and accessibility? The Covid-19 pandemic accelerated streaming services becoming the dominant form of entertainment. There are a handful of new platforms, each with thousands of hours of content, but not much change or differentiation in the user journeys. For the most part, everywhere from Netflix to illegal streaming platforms use similar video streaming UX standards, and...

    Continue Reading

  • Innovation with collaborationExperimenting with AI and investigative journalism in the Americas.

    Lee este artículo en español. How might we use AI technologies to innovate newsgathering and investigative reporting techniques? This was the question we posed to a group of seven newsrooms in Latin America and the US as part of the Americas Cohort during the 2021 JournalismAI Collab Challenges. The Collab is an initiative that brings together media organizations to experiment with AI technologies and journalism. This year,  JournalismAI, a project of Polis, the journalism think-tank at...

    Continue Reading

  • Innovación con colaboraciónCuando el periodismo de investigación experimenta con inteligencia artificial.

    Read this article in English. ¿Cómo podemos usar la inteligencia artificial para innovar las técnicas de reporteo y de periodismo de investigación? Esta es la pregunta que convocó a un grupo de siete organizaciones periodísticas en América Latina y Estados Unidos, el grupo de las Américas del 2021 JournalismAI Collab Challenges. Esta iniciativa de colaboración reúne a medios para experimentar con inteligencia artificial y periodismo. Este año, JournalismAI, un proyecto de Polis, la think-tank de periodismo...

    Continue Reading

  • AI, Automation, and Newsrooms: Finding Fitting Tools for Your Organization

    If you’d like to use technology to make your newsroom more efficient, you’ve come to the right place. Tools exist that can help you find news, manage your work in progress, and distribute your content more effectively than ever before, and we’re here to help you find the ones that are right for you. As part of the Knight Foundation’s AI for Local News program, we worked with the Associated Press to interview dozens of......

    Continue Reading

Storytelling Tools

We build easy-to-use tools that can help you tell better stories.

View More