Attend an NYC Open Data Training, Realize Possibilities!

This post’s big takeaway: NYC has more data and dashboards at your fingertips than you know. Go gettum and build.

On October 19, Ellie and I both attended a training on NYC’s Open Data portal, a collection of datasets provided by city agencies. This training is an ongoing collaboration between NYC Open Data and BetaNYC.

You can sign up for your own 90-minute session here, which we highly recommend; our training was expertly led by Nathan Storey, who took us through a lot of useful context before digging into the data portal itself. What it covers, from the training website:

• What is NYC Open Data

• History of the NYC Open Data program

• How to frame questions for working with NYC Open Data

• Using the NYC Open Data website, filtering, and visualizing * datasets

• Useful tools powered by NYC Open Data

If you’re wondering where NYC Open Data is situated in the government, look no further than its overview page. It’s within the Mayor’s Office of Technology and Innovation, which is headed by the city’s (mayor-selected) Chief Technology Officer, Matthew Fraser:

NYC Open Data is managed by the Open Data Team at the NYC Office of Technology and Innovation (OTI). The team works with City agencies to identify and make data available, coordinate platform operations and improvements, and promote the use of Open Data both within government and throughout NYC. Each City agency also has an Open Data Coordinator, who serves as the main point of contact for the Open Data team and the public, and works to identify, document, structure, and manage the agency’s public datasets.

Here are some brief notes from the training, which we might update and polish later. In the meantime, attend your own training and bring your friends.

History of Open Data

Late 19th and early 20th century:

• In the Progressive Era, the City Record was implemented (a record of everything the city does). It was the first instance of “Open Data,” and it still exists today. If you want to know about public meetings, contracts, and more, you’ll find it in the City Record.

1960s-1970s:

• Freedom of Information laws are passed around the country - info available by request

  • People can have access to internal government information, but you have to know what you’re looking for (and you have to know that there’s something there to look for)

• Passed at state and federal level. NYS was 1974

• FOIL and FOIA (NYS and federal laws, respectively)

1993

• Public Data Directory created for NYC

  • Created to list all the datasets that were created by all NYC’s agencies

  • Published by the Commission for Public Information and Communication

  • Created the concept of a public liaison for each agency

  • Attempted to expand transparency, and was a move toward exploring data rather than having to know exactly what you’re looking for

  • Some of these datasets and systems are still being used

2012

• NYC Open Data Law signed - public by default, rather than public by request

  • Required that all datasets created by agencies be public by default, unless there is a compelling reason not to, like personal information

  • Paradigm shift: you don’t need to know exactly what you’re looking for, you can just have general access

• NYC was a pioneer here—many others have followed since

• Over one million visitors each year to NYC Open Data portal

  • More than 3,000 datasets

  • Made possible by a network of approximately 100 Open Data Coordinators across agencies, offices, and commissions

Data Acquisition and Manipulation

Getting and manipulating data in the Open Data portal is straightforward enough, especially if you’re already used to doing data stuff in Excel, Python, etc. But even if you’re not, you’ll be fine after this training.

Plus, there’s a lot of cool things to look up, and BetaNYC has a lot of cool tools and info that take NYC Open Data’s work even further.

As an example, Ellie quickly pulled 311 call data and grouped it by the responding government agency (see the pie chart). She then drilled down into calls sent specifically to the Department of Health and Mental Hygiene, and looked at call locations for one example day last weekend (see map). This kind of data pull/viz can be done quickly, but so much more can be done as well.

Conclusions:

• Sign up for your own training here.

• Tell your friends. Ideally attend with them.

• Realize we have a lot of cool data and tools at our fingertips to help make the city better.

• Join BetaNYC! They’re very friendly, and they have an Open Data Week every year—so watch out for 2023 details. You can present your projects, and see what others have come up with.