Summary & Our Definition of Fairness

Before delving into the data and running any analysis, we determined that our metric for fairness would be based on “equal-opportunity” of being searched, i.e. the question we wanted to answer was: do people who live in certain areas or look a certain way more likely to be searched with no arrest? By certain areas, we mean the population breakdown of precincts, i.e. if a neighbourhood is more skewed towards a certain demographic, does it impact the likelihood of a stop and frisk?

“Disparate impact” in the U.S. Labor Law refers to activities that adversely affect a group of people of a protected characteristic (say race). This can be applied to fairness in stop-and-frisk, i.e. do people of a protected characteristic get more adversely impacted with stop-and-frisk, i.e. are there a lower number of hits?

We decided to break it down by precinct as opposed to census block, as that’s how the NYPD bureaus are divided. The fairness question that this analysis raises is with respect to the police: were the stops fair? And, if not, which precincts were particularly unfair?

Initial Investigation

Initially, before running the linear model, we ran through some basic number crunching to see if there was any evidence of unfairness. Here, we looked at two things:

•  Figure 1 shows the hit ratio for violent crimes, by precinct, across blacks, whites, black Hispanics and white Hispanics. Here, we saw that the (twelve) lowest hit ratios were distributed between Black Hispanics (4), Whites (7) and a single white Hispanic.
  Further investigation suggested that, for the most part, the hit ratio was at around the 0.05 mark. Interestingly, for some precincts (20, 32, 42, 45, 48), where the hit ratio for non-whites was very low, the hit ratio for Whites was significantly higher.
  On the other hand, in some precincts (50, 76, 78), the hit ratio for blacks was much higher than the hit ratio for whites.
  Effectively, there was no evidence of unfairness across the board, but if we narrowed it down by precinct, there was evidence of bias.

• Figure 2.1, 2.2 and 2.3 plot the hit ratios for blacks against precincts with different racial backgrounds. The idea behind this was: if there is racial bias in a precinct, then the hit rate for blacks in that community will be low. Or, put another way, are people more suspicious of a minority community in their midst for no good reason (which would explain the low hit rate)?
  Here, we found that in a predominantly white precinct (where 80-100% of the population was white), the hit ratio for blacks was at 0.11, whereas at a precinct where there were less than 20% whites, the number was at 0.02. But, the hit ratio for whites was, for the most part, constant ~0.08.
  In neighbourhoods that were predominantly Hispanic, on the other hand (60-80%), the hit ratio was low at 0.01. Here, the hit rate for whites was ~0.065. However, when there were less than 20% Hispanics, the hit ratio exceeded 0.07. And, for whites, it exceeded 0.08.
  Finally, in the hit ratio was high (exceeded 0.08) in precincts where the black population was sub-20% but low (0.01) where over 80% of the population was black.
  While these was interesting to note, the numbers weren’t outlandishly skewed, and consequently, there was no obvious bias evident.

Method

We ran two regressions – one on race and one on the likelihood of being detained for a reported interaction in precincts based on the racial makeup of that precinct. The precinct racial makeup data was taken from all NYC2010 census blocks with the precinct, modified by John Keefe.

Race coefficients:

First, we took a look at the differences between race after applying random effects to sex, precinct and age. Our findings show that when looking at all crime, there is not much difference between whites and blacks. Both were slightly more likely than all other races but one (P and Q together form the Hispanic group) to produce a hit. This seems to show that both whites and blacks are policed more reasonably than other races because they produce fewer stops without arrests.

However, when you remove all non-violent crime, whites are the only race where the odds of a stop producing a hit increases. All the heavily represented races (lower std. deviation) converge to zero when we remove non-violent crime, showing that there’s much less disparity. Yet, there still appears to be some racial discrimination toward non-whites.

Precinct coefficients:

By running a regression on the percentage of each precinct that was of a given race for black, white and Hispanic, we hoped to be able to identify any racial disparities between policing in particular precincts.

Both the violent-only and full set of data seem to show significant differences between whites and blacks. When factoring in non-violent crime, it appears black communities have many more stops without arrests than either the white or the Hispanic communities.

Our findings also seem to show predominantly white communities could use more policing to cut down on violent crime, as stopping a person in a white precinct increases the chances of a hit when only looking at violent crime. However, black and Hispanic communities are pretty much on par with each other with violent-only crime.

Residual effects:

We can see that if we breakdown our LME model by hit ratio vs. the precinct (with random effects on precinct), we get a fairly straight line, indicating that we get a fairly reliable predictor using just the precinct.

What is interesting, though, is that this doesn’t change significantly when we add race to the mix, indicating that using the precinct in isolation doesn’t decrease the reliability of the predictor compared to using a combination of the precinct and race.

There would be no disparate impact if the probability of hits given race is equivalent to the probability of hits when race is absent (i.e. P(hits | race == 1) ≈ P(hits | race == 0)). We can see that this is the case here.

Final thoughts 

For the most part, through this exercise, we didn’t see a lot of bias in policing. There is evidence that some precinct are more biased – or more unfair – than others, using our definition of “unfairness”, but not enough where we can claim there is systematic bias.

The residual effects chart is probably the most interesting, as it seems to indicate that throwing race into the mix doesn’t improve predictability – but, that could be explained by bias already being built into the precinct. Or, because, policing had become more careful in the early 2010s after the accusations of bias in the mid-to-late noughties.

Here’s a tutorial on the steps I went through to generate the data for Assignment 3, statistical modeling of stop & frisk data with precinct as the unit of analysis. The first problem is adding a precinct ID to every stop and frisk event, which means matching stop locations to precinct areas. To get baseline racial makeup we also need to sum census data over precincts.

Thanks to John Keefe of WNYC for valuable data and advice.

Just the data

• Here is NYPD’s 2011 stop & frisk data, with appended precinct and lat/long. The column names are explained here.
• Here are all all NYC 2010 census blocks, with appended precinct, from John Keefe, and here is the census data dictionary (see pp.6-10)

How to make this data

1. First, download NYPD raw stop and frisk data here. I used 2011, since that was the year with the most stops before the program was drastically curtailed. There’s also documentation on that page, which explains the data format

2. The XCOORD and YCOORD columns record the position of each stop, in the “New York-Long Island State Plane Coordinate System” also known as EPSG 2908. To do anything useful with this, we’re going to need to convert into lat/long using QGis. Use Layer -> Add Layer -> Add Delimited Text Layer to load the CSV file. It should automatically guess the correct column names. When asked, pick EPS 2908 for the coordinate system. If all goes well, you are now looking at this:

3. We need to save this in a more standard coordinate system for comparison to precinct boundaries. Right click on the “2011” layer and Save As as an ESRI Shapefile with CRS (“coordinate reference system”) of WGS84, which should also be the “Project CRS.” Ensure “add saved file to map” is checked. Now you’ve got a new layer in standard GPS-style lat/long coordinates.

4. Now we need to know the geographic outlines for each police precinct. The raw shape files are here, or your can look at them in a Fusion Table here. In QGIS, Use Add Layer -> Add Vector Layer and select the precinct shapefile (nypp_4326.zip).

5. To assign points to their precinct, run Add polygon attributes to point in the data processing toolbox (Processing -> Toolbox). The “Precinct” field should appear automatically. Again you’ll have a new layer. Right click and save that as a CSV. You should end up with a file that is exactly the same as the NYPD’s 2011 CSV, but with lat/long and precinct columns added.

6. Keefe has also done the work of enriching New York City census blocks with precinct keys, which he’s put in this this  Fusion Table. The data above is a CSV download of this file.

 

For this exercise, you will break into three groups. Two will analyze data, and the third will do legal research. The data for this assignment is available here. All groups will turn in their assignment by posting on this blog — you’ll have logins shortly — and linking to a github repository of your code.

All groups: How can we encode our ideas about racial fairness into a quantitative metric? That is the fundamental question underlying this assignment and you must answer it. Don’t build models that give you answers to useless questions; each model you build must embody some justifiable concept of fairness. Many such metrics have been proposed; part of the assignment is researching and evaluating them. I’ve proposed some models that might be interesting, but I’ll be just has happy — perhaps happier — to have you tell me why these particular model formulations will not yield an interesting result. Also, you have to tell me what your modeling results mean. Is there bias? In what way, how significant is it, and what are alternate explanations? Uninterpreted results will not get a passing grade.

Group 1: Analyze the stop and frisk data using a multi-level linear model

This group will analyze the data along the lines of Gelman 2006. However, that paper used Bayesian estimation whereas this team will used standard linear regression. Use RStudio with the lme4 package, as described in this tutorial.

As above, you need to choose and justify quantitative definitions of fairness. Should you look at stop rate? Hit rate? And should you compare to the racial composition of each precinct? Or per-race crime rates? And if so, which crimes?

Each conceptual definition of fairness can be embodied in many different statistical models. Compare at least two different statistical formulations. For example you might end up modeling:

• hit rate vs race, per precinct
• hit rate vs race, per precinct, with precinct random effects

You must also choose the unit of analysis. Is precinct or census block a better unit? Why? Or you could compare the same model on different units.

Your final post must include a justification of the metric and model choices, and useful visualization of the results, and interpretation of both the regression coefficients and uncertainty measures (standard errors on regression coefficients, and modeling residuals.)

Group 2: Analyze the stop and frisk data using a Bayesian model

This group will analyze the data along the lines of Simoiu 2016. For a tutorial on how to set up Bayesian modeling in R, see Bayesian Linear Mixed Models using STAN.

As with group 2, you must research and choose a definition of fairness, fit at least two different statistical formulations, and interpret your results including the uncertainty (posterior distributions and residuals.) Be sure to visualize your model fit, as in figure 7 of Simoiu.

I would be happy to see you replicate the threshold test of Simoiu. However, I want you to explain why the threshold test makes sense as a fairness metric. If it doesn’t make sense I want you to design a new model. Perhaps the assumption that each officer can make correctly calibrated estimates of the probability that someone is carrying contraband is unrealistic, and your model should be based on the idea that the estimates are biased and try to model that bias as latent variables.

Group 3: Research legal and policy precedent for statistical tests

This group will scour the legal literature to determine what sorts of statistical tests have been used, or could be used, to answer legal questions of discrimination. You will also research the related policy literature: what sorts of tests have governments, companies, schools etc. used to evaluate the presence or significance of discrimination. For an entry into the literature, you could do worse than to start with the works referenced by Big Data’s Disparate Impact.

This group will not be coding, but I expect you to ask not only what fairness metrics might be appropriate (as the other groups must also do) but 1) whether or not these metrics might hold up in court and 2) whether they have ever been used outside of court.

And one particular question I would like you to answer: would Simiou’s “threshold test” have legal or policy relevance?

Submitting your Results

All groups will report their results on this blog, and present their findings to the class on Friday, November 4.

The course is a hands-on introduction to the areas of computer science that have a direct relevance to journalism, and the broader project of producing an informed and engaged public. We will touch on many different technical and social topics: information recommendation systems but also filter bubbles, principles of statistical analysis but also the human processes which generate data, network analysis and its role in investigative journalism, visualization techniques and the cognitive effects involved in viewing a visualization. Assignments will require programming in Python, but the emphasis will be on clearly articulating the connection between the algorithmic and the editorial.

Our scope is wide enough to include both relatively traditional journalistic work, such as computer-assisted investigative reporting, and the broader information systems that we all use every day to inform ourselves, such as search engines and social media. The course will provide students with a thorough understanding of how particular fields of computational research relate to journalism practice, and provoke ideas for their own research and projects.

Research-level computer science material will be discussed in class, but the emphasis will be on understanding the capabilities and limitations of this technology. Students with a CS background will have opportunity for algorithmic exploration and innovation, however the primary goal of the course is thoughtful, application-oriented research and design.

Format of the class, grading and assignments.
This is a fourteen week course for Masters’ students which has both a six point and a three point version. The six point version is designed for CS & journalism dual degree students, while the three point version is designed for those cross listing from other schools. The class is conducted in a seminar format. Assigned readings and computational techniques will form the basis of class discussion. The course will be graded as follows:

• Assignments: 80%. There will be a homework assignment after most classes.
• Class participation: 20%

Assignments will be completed in groups (except dual degree students, who will work individually) and involve experimentation with fundamental computational techniques. Some assignments will require intermediate level coding in Python, but the emphasis will be on thoughtful and critical analysis. As this is a journalism course, you will be expected to write clearly.

Dual degree students will also have a final project. This will be either a research paper, a computationally-driven story, or a software project. The class is conducted on pass/fail basis for journalism students, in line with the journalism school’s grading system. Students from other departments will receive a letter grade.

Week 1: Introduction and Clustering – 9/16
Slides
First we ask: where do computer science and journalism intersect? CS techniques can help journalism in four different areas: data-driven reporting, story presentation, information filtering, and effect tracking. Then we jump right into high dimensional data analysis and visualization, which we’ll need to study filtering, with an example of clustering, visualizing, and interpreting feature vectors of voting patterns.

Required

• What should the digital public sphere do?, Jonathan Stray
• Computational Journalism, Cohen, Turner, Hamilton

Recommended

• Precision Journalism, Ch.1, Journalism and the Scientific Tradition, Philip Meyer

Viewed in class

• What did private security contractors do in Iraq and document mining methodology, Jonathan Stray
• The Jobless rate for People Like You, New York Times
• Surgeon Scorecard, ProPublica
• Message Machine, ProPublica
• Using clustering to analyze the voting blocs in the UK House of Lords

Unit 1: Filtering

Week 2: Text Analysis – 9/23
Slides
Can we use machines to help us understand text? In this class we will cover basic text analysis techniques, from word counting to topic modeling. The algorithms we will discuss this week are used in just about everything: search engines, document set visualization, figuring out when two different articles are about the same story, finding trending topics. The vector space document model is fundamental to algorithmic handling of news content, and we will need it to understand how just about every filtering and personalization system works.

Required

• TF-IDF is about what matters, Aaron Schumacher
• Topic modeling by hand, Shawn Graham
• How ProPublica’s Message Machine reverse engineers political microtargeting, Jeff Larson

Recommended

• A full-text visualization of the Iraq war logs, Jonathan Stray
• Introduction to Information Retrieval Chapter 6, Scoring, Term Weighting, and The Vector Space Model, Manning, Raghavan, and Schütze.
• Probabilistic Topic Models, David M. Blei

Examples

• Watchwords: Reading China Through its Party Vocabulary, Qian Gang

Assignment:  LDA analysis of State of the Union speeches.

Week 3: Filtering algorithms
Slides
This week we begin our study of filtering with some basic ideas about its role in journalism. We will study the details of several algorithmic filtering approaches including Twitter, Reddit’s comment ranking, the Newsblaster system  (similar to Google News) and the New York Times recommendation engine.

Required

• Building the Next New York Times Recommendation Engine, Alexander Spangher
• Tracking and summarizing news on a daily basis with Columbia Newsblaster, McKeown et al

Recommended

• Matrix Factorization Techniques for Recommender Systems, Koren et al
• How Reddit Ranking Algorithms Work, Amir Salihefendic
• Google News Personalization: Scalable Online Collaborative Filtering, Das et al
• The Hermeneutics of Screwing Around, or What You Do With a Million Books, Stephen Ramsay

Discussed in class

• Item-Based Collaborative Filtering Recommendation Algorithms, Sarwar et. al

Week 4: Filters as Editors 
Slides
We’ve seen what filtering algorithms do, but what should they do? This week we’ll study the social effects of filtering system design, how Google Search and other systems are optimized in practice, and start to ask about possible ill effects like polarization and fake news.
Required

• How Facebook’s Foray into Automated News Went from Messy to Disastrous, Will Oremus
• Who should see what when? Three design principles for personalized news Jonathan Stray

Recommended

• How does Google use human raters in web search?, Matt Cutts
• A Group is its own worst enemy, Clay Shirky
• What Happens to #Ferguson Affects Ferguson: Net Neutrality, Algorithmic Filtering and Ferguson, Zeynep Tufekci
• Guess what? Automated news doesn’t quite work, Gabe Rivera
• Can an algorithm be wrong?, Tarleton Gillespie

Viewed in class

• Israel, Gaza, War & Data: social networks and the art of personalizing propaganda, Gilad Lotan
• Learning from Stackoverflow, first fifteen minutes, Joel Spolsky
• What is Twitter, a Social Network or a News Media?, Haewoon Kwak, et al,

Assignment – Design a filtering algorithm for an information source of your choosing

Unit 2: Interpreting Data

Week 5: Quantification, Counting, and Statistics
Slides
Every journalist needs a basic grasp of statistics. Not t-tests and all of that, but more grounded. Where does data come from at all? How do we know we’re measuring the right thing, and measuring it properly? Then a solid understanding of the concepts that come up most in journalism: relative risk, conditional probability, the regressions and control variables, the use of statistical models generally. Finally, the state of the art in data-driven tests for discrimination.

• The Curious Journalist’s Guide to Data: Quantification, Jonathan Stray
• Testing for Racial Discrimination in Police Searches of Motor Vehicles, Simoiu et al

Recommended

• Why Not to Trust Statistics, Ben Orlin
• Statistics for Decision Makers: Base Rate Fallacy, Bernard Szlachta
• The Quartz Guide to Bad Data, Christopher Groskopf
• Solve Every Statistics Problem with One Weird Trick, Jonathan Stray
• Operationalizing, or the function of measurement in modern literary theory, Franco Moretti
• The Curious Journalist’s Guide to Data: Prediction, Jonathan Stray

Week 6: Drawing conclusions from data 
Slides
This week is all about using data to report on ambiguous, complex, charged issues. It’s incredibly easy to fool yourself, but fortunately, there is a long history of fields grappling with the problem of determining truth in the face of uncertainty, from statistics to intelligence analysis. This week includes: statistical testing and statistical significance, Bayesianism in theory and practice, determining causality, p-hacking and reproducibility, analysis of competing hypothesis.

Required

• The Curious Journalist’s Guide to Data: Analysis, Jonathan Stray
• I Fooled Millions Into Thinking Chocolate Helps Weight Loss. Here’s How, John Bohannon,

Recommended

• Why most published research findings are false, John P. A. Ioannidis
• If correlation doesn’t imply causation, then what does?, Michael Nielsen
• The Psychology of Intelligence Analysis, chapter 8. Richards J. Heuer
• The Introductory Statistics Course: a Ptolemaic Curriculum, George W. Cobb

Viewed in class

• Science Isn’t Broken, Christie Aschwanden
• Correlation and causation, Business Insider

Assignment: Analyze NYPD stop and frisk data for racial discrimination. Details here.

Week 7: Algorithmic Accountability 
Slides
Our society is woven together by algorithms. From high frequency trading to predictive policing, they regulate an increasing portion of our lives. But these algorithms are mostly secret, black boxes form our point of view. We’re at they’re mercy, unless we learn how to interrogate and critique algorithms. We’ll focus in depth on analysis of discrimination of various types, and how this might (or might not) be possible in computational journalism.

Required

• How Uber surge pricing really works, Nick Diakopoulos
• How Big Data is Unfair, Moritz Hardt

Recommended

• How the Journal Tested Prices and Deals Online, Jeremy Singer-Vine, Ashkan Soltani and Jennifer Valentino-DeVries
• How We Analyzed the COMPAS Recidivism Algorithm, Larson et al.
• Big Data’s Disparate Impact, Barocas and Selbst
• Algorithmic Accountability: On the investigation of black boxes, Nick Diakopoulos
• How Algorithms Shape our World, Kevin Slavin

Unit 3: Methods

Week 8: Visualization 
Slides
An introduction into how visualization helps people interpret information. Design principles from user experience considerations, graphic design, and the study of the human visual system. The Overview document visualization system used in investigative journalism.

Required

• Visualization, Tamara Munzner

Recommended

• 39 Studies about Human Perception in 30 minutes, Kennedy Elliot
• Computational Information Design chapters 1 and 2, Ben Fry
• Overview: The Design, Adoption, and Analysis of a Visual Document Mining Tool For Investigative Journalists, Brehmer et al.
• Journalism in an age of data, Geoff McGhee
• Visualization Rhetoric: Framing Effects in Narrative Visualization, Hullman and Diakopolous

Week 9: Knowledge representation 
Slides
How can journalism benefit from encoding knowledge in some formal system? Is journalism in the media business or the data business? And could we use knowledge bases and inferential engines to do journalism better? This gets us deep into the issue of how knowledge is represented in a computer. We’ll look at traditional databases vs. linked data and graph databases, entity and relation detection from unstructured text, and traditional both probabilistic and propositional formalisms. Plus: NLP in investigative journalism, automated fact checking, and more.

Required

• What do journalists do with documents? Jonathan Stray
• Relation extraction and scoring in DeepQA – Wang et al, IBM

Recommended

• A fundamental way newspaper websites need to change, Adrian Holovaty
• The State of Automated Fact Checking, Full Fact
• Storylines as Data in BBC News, Jeremy Tarling
• Building Watson: an overview of the DeepQA project
• Automation in the newsroom, Celeste LeCompte.

Viewed in class

• The next web of open, linked data – Tim Berners-Lee TED talk
• https://schema.org/NewsArticle
• Connected China, Reuters/Fathom

Assignment: Text enrichment experiments using StanfordNER entity extraction.

Week 10: Network analysis 
Slides
Network analysis (aka social network analysis, link analysis) is a promising and popular technique for uncovering relationships between diverse individuals and organizations. It is widely used in intelligence and law enforcement, but not so much in journalism. We’ll look at basic techniques and algorithms and try to understand the promise — and the many practical problems.

Required

• Analyzing the Data Behind Skin and Bone, ICIJ
• Simmelian Backbones: Amplifying Hidden Homophily in Facebook Networks. A soophisticated and sociologically-aware network analysis method.

Recommended

• Visualizing Communities, Jonathan Stray
• Identifying the Community Power Structure, an old handbook for community development workers about figuring out who is influential by very manual processes.
• The network of global corporate control, Vitali et. al.
• The Dynamics of Protest Recruitment through an Online Network, Sandra González-Bailón, et al.
• Sections I and II of Community Detection in Graphs, Fortunato
• Centrality and Network Flow, Borgatti
• Exploring Enron, Jeffrey Heer

Examples:

• Galleon’s Web, Wall Street Journal
• Muckety
• Theyrule.net,
• Who Runs Hong Kong?, South China Morning Post

Assignment: Compare different centrality metrics in Gephi.
Week 11: Privacy, Security, and Censorship 
Slides
Who is watching our online activities? How do you protect a source in the 21st Century? Who gets to access to all of this mass intelligence, and what does the ability to survey everything all the time mean both practically and ethically for journalism? In this lecture we will talk about who is watching and how, and how to create a security plan using threat modeling.

Required

• Digital Security for Journalists, Part 1 and Part 2, Jonathan Stray
• Hearst New Media Lecture 2012, Rebecca MacKinnon

Recommended

• CPJ journalist security guide section 3, Information Security
• Global Internet Filtering Map, Open Net Initiative
• Unplugged: The Show part 9: Public Key Cryptography
• Diffe-Hellman key exchange, ArtOfTheProblem
• Tor Project Overview
• Who is harmed by a real-names policy, Geek Feminism

Assignment: Use threat modeling to come up with a security plan for a given scenario.

Week 12: Tracking flow and impact 

How does information flow in the online ecosystem? What happens to a story after it’s published? How do items spread through social networks? We’re just beginning to be able to track ideas as they move through the network, by combining techniques from social network analysis and bioinformatics.

Required

• Metrics, Metrics everywhere: Can we measure the impact of journalism?, Jonathan Stray
• Meme-tracking and the Dynamics of the News Cycle, Leskovec et al.
• How promotion affects pageviews on the New York Times website, Brian Abelson

Recommended

• NewsLynx: A Tool for Newsroom Impact Measurement, Michael Keller, Brian Abelson
• The role of social networks in information diffusion, Eytan Bakshy et al.
• Defining Moments in Risk Communication Research: 1996–2005, Katherine McComas
• Chain Letters and Evolutionary Histories, Charles H. Bennett, Ming Li and Bin Ma
• Competition among memes in a world with limited attention, Weng et al.
• Zach Seward, In the news cycle, memes spread more like a heartbeat than a virus
• How hidden networks orchestrated Stop Kony 2012, Gilad Lotan

 Final projects due 12/31  (dual degree Journalism/CS students only)