♪ ♪
SARAH BRAYNE: We live in this era where
we leave digital traces throughout the course
of our everyday lives.
ANDY CLARNO: What is this data,
how is it collected, how is it being used?
NARRATOR: One way it's being used is to make predictions
about who might commit a crime...
Hey, give me all your money, man!
NARRATOR: ...and who should get bail.
JUDGE: On count one, you're charged with felony intimidation...
ANDREW FERGUSON: The idea is that if you look at past crimes,
you might be able to predict the future.
WILLIAM ISAAC: We want safer communities,
we want societies that are less incarcerated.
NARRATOR: But is that what we're getting?
Are the predictions reliable?
CATHY O'NEIL: I think algorithms can,
in many cases, be better than people.
But, of course, algorithms don't have consciousness.
The algorithm only knows what it's been fed.
RUHA BENJAMIN: Because it's technology,
we don't question them as much as we might
a racist judge or a racist officer.
They're behind this veneer of neutrality.
ISAAC: We need to know who's accountable
when systems harm the communities
that they're designed to serve.
NARRATOR: Can we trust the justice of predictive algorithms?
And should we?
"Computers Vs. Crime,"
right now, on "NOVA."
(computers booting up)
♪ ♪
NARRATOR: We live in a world of big data,
where computers look for patterns
in vast collections of information
in order to predict the future.
And we depend on their accuracy.
Is it a good morning for jogging?
Will this become cancer?
What movie should I choose?
The best way to beat traffic?
Your computer can tell you.
Similar computer programs, called predictive algorithms,
are mining big data to make predictions
about crime and punishment--
reinventing how our criminal legal system works.
Policing agencies have used these computer algorithms
in an effort to predict where the next crime will occur
and even who the perpetrator will be.
ASSISTANT DISTRICT ATTORNEY: Here, the state is recommending...
NARRATOR: Judges use them
to determine who should get bail
and who shouldn't.
JUDGE: If you fail to appear next time, you get no bond.
NARRATOR: It may sound like the police of the future
in the movie "Minority Report." I'm placing you under arrest
for the future murder of Sarah Marks.
NARRATOR: But fiction it's not.
How do these predictions actually work?
Can computer algorithms
make our criminal legal system more equitable?
Are these algorithms truly fair and free of human bias?
ANDREW PAPACHRISTOS: I grew up in Chicago
in the 1980s and early 1990s.
♪ ♪
My dad was an immigrant from Greece,
we worked in my family's restaurant,
called KaMar's.
NARRATOR: Andrew Papachristos was a 16-year-old kid
in the North Side of Chicago in the 1990s.
I spent a lot of my formative years busing tables,
serving people hamburgers and gyros.
It kind of was a whole family affair.
NARRATOR: Young Papachristos was aware the streets could be dangerous,
but never imagined the violence would touch him or his family.
REPORTER: Two more gang-related murders Monday night.
PAPACHRISTOS: And of course, you know, the '80s and '90s in Chicago
was some of the historically most violent periods in Chicago.
Street corner drug markets, street organizations.
And then like a lot of other businesses on our, on our block
and in our neighborhood,
local gangs tried to extort my family and the business.
And my dad had been running KaMar's for 30 years
and kind of just said no.
♪ ♪
(sirens blaring)
NARRATOR: Then, one night, the family restaurant burned to the ground.
Police suspected arson.
PAPACHRISTOS: It was quite a shock to our family,
'cause everybody in the neighborhood worked
in the restaurant at one point in their life.
And my parents lost 30 years of their lives.
That was really one of the events that made me want to
understand violence.
Like, how could this happen?
♪ ♪
NARRATOR: About a decade later,
Papachristos was a graduate student searching for answers.
PAPACHRISTOS: In graduate school,
I was working on a violence prevention program
that brought together community members,
including street outreach workers.
And we were sitting at a table,
and one of these outreach workers asked me,
the university student,
"Who's next?
Who's going to get shot next?"
And where that led was me sitting down
with stacks of shooting and, and homicide files
with a red pen and a legal pad,
by hand creating these network images
of, this person shot this person,
and this person was involved with this group and this event,
and creating a web of these relationships.
And then I learned that
there's this whole science about networks.
I didn't have to invent anything.
♪ ♪
NARRATOR: Social network analysis was already influencing
popular culture.
"Six Degrees of Separation" was a play on Broadway.
And then, there was Six Degrees of Kevin Bacon.
PAPACHRISTOS: The idea was, you would play this game,
and whoever got the shortest distance to Kevin Bacon
would win.
So Robert De Niro was in a movie with so-and-so,
who was in a movie with Kevin Bacon.
It was creating, essentially, a series of ties
among movies and actors.
And in fact, there's a mathematics
behind that principle.
It's actually old mathematical graph theory, right?
That goes back to 1900s mathematics.
And lots of scientists started seeing that there were
mathematical principles,
and computational resources-- computers, data--
were at a point that you could test those things.
So it was in a very exciting time.
We looked at arrest records, at police stops,
and we looked at victimization records.
Who was the victim of a homicide
or a non-fatal shooting?
♪ ♪
The statistical model starts by creating the social networks of,
say, everybody who may have been arrested
in a, in a particular neighborhood.
So Person A and Person B were in a robbery together,
they have a tie, and then Person B and Person C
were, were stopped by the police in another instance.
And it creates networks of thousands of people.
Understanding that events are connected,
places are connected.
That there are old things, like disputes between crews,
which actually drive behavior for generations.
What we saw was striking.
(snaps): And you could see it immediately,
and you could see it a mile away.
Which was, gunshot victims clumped together.
You, you very rarely see one victim.
You see two, three, four.
Sometimes they string across time and space.
And then the model predicts, what's the probability
that this is going to lead to a shooting
on the same pathway in the future?
(gun firing, people shouting)
REPORTER: Another young man lies dead.
NARRATOR: In Boston, Papachristos found
that 85% of all gunshot injuries
occurred within a single social network.
Individuals in this network
were less than five handshakes away
from the victim of a gun homicide
or non-fatal shooting.
The closer a person was
connected to a gunshot victim, he found,
the greater the probability that that person would be shot.
Around 2011, when Papachristos was presenting
his groundbreaking work on social networks
and gang violence,
the Chicago Police Department wanted to know more.
PAPACHRISTOS: We were at a conference.
The then-superintendent of the police department,
he was asking me a bunch of questions.
He had clearly read the paper.
NARRATOR: The Chicago Police Department
was working on its own predictive policing program
to fight crime.
They were convinced that Papachristos's model
could make their new policing model even more effective.
LOGAN KOEPKE: Predictive policing involves
looking to historical crime data to predict future events,
either where police believe crime may occur
or who might be involved in certain crimes.
♪ ♪
So it's the use of historical data to forecast a future event.
NARRATOR: At the core of these programs is software,
which, like all computer programs,
is built around an algorithm.
So, think of an algorithm like a recipe.
♪ ♪
You have inputs,
which are your ingredients, you have the algorithm,
which is the steps.
♪ ♪
And then there's the output,
which is hopefully the delicious cake you're making.
GROUP: Happy birthday!
So one way to think about algorithms is to think about
the hiring process.
In fact, recruiters have been studied for a hundred years.
And it turns out many human recruiters
have a standard algorithm
when they're looking at a risumi.
So they start with your name, and then they look to see
where you went to school, and then finally,
they look at what your last job was.
If they don't see the pattern they're looking for...
(bell dings) ...that's all the time you get.
And in a sense, that's exactly what artificial intelligence
is doing, as well, in a very basic level.
It's recognizing sets of patterns and using that
to decide what the next step in its decision process would be.
♪ ♪
NARRATOR: What is commonly referred to
as artificial intelligence, or A.I.,
is a process called machine learning,
where a computer algorithm will adjust on its own,
without human instructions,
in response to the patterns it finds in the data.
These powerful processes can analyze more data
than any person can,
and find patterns never recognized before.
The principles for machine learning
were invented in the 1950s,
but began proliferating only after about 2010.
What we consider machine learning today
came about because hard drives became very cheap.
So it was really easy to get a lot of data on everyone
in every aspect of life.
And the question is, what can we do with all of that data?
Those new uses are things like predictive policing,
they are things like deciding whether or not a person's
going to get a job or not,
or be invited for a job interview.
NARRATOR: So how does such a powerful tool like machine learning work?
Take the case of a hiring algorithm.
First, a computer needs to understand the objective.
Here, the objective is identifying
the best candidate for the job.
The algorithm looks at risumis of former job candidates
and searches for keywords in risumis of successful hires.
The risumis are what's called training data.
The algorithm assigns values to each keyword.
Words that appear more frequently in the risumis
of successful candidates are given more value.
The system learns from past risumis the patterns
of qualities that are associated with successful hires.
Then it makes its predictions by identifying these
same patterns from the risumis of potential candidates.
♪ ♪
In a similar way,
the Chicago police wanted to find patterns in crime reports
and arrest records to predict who would be connected
to violence in the future.
They thought Papachristos's model could help.
Obviously we wanted to, and tried, and framed and wrote
all the caveats and made our recommendations to say,
"This research should be in this public health space."
But once the math is out there,
once the statistics are out there,
people can also take it and do what they want with it.
NARRATOR: While Papachristos saw the model as a tool to identify
future victims of gun violence,
CPD saw the chance to identify not only future victims,
but future criminals.
First it took me, you know, by, by surprise,
and then it got me worried.
What is it gonna do?
Who is it gonna harm?
♪ ♪
NARRATOR: What the police wanted to predict was who was at risk
for being involved in future violence.
Gimme all your money, man.
NARRATOR: Training on hundreds of thousands of arrest records,
the computer algorithm looks for patterns or factors
associated with violent crime
to calculate the risk that an individual
will be connected to future violence.
Using social network analysis,
arrest records of associates
are also included in that calculation.
The program was called the Strategic Subject List, or SSL.
It would be one of the most controversial
in Chicago policing history.
ANDY CLARNO: The idea behind the Strategic Subjects List,
or the SSL,
was to try to identify the people who would be
most likely to become involved as what they called
a "party to violence," either as a shooter or a victim.
♪ ♪
NARRATOR: Chicago police would use Papachristos's research
to evaluate what was called an individual's
"co-arrest network."
And the way that the Chicago Police Department
calculated an individual's network was through
kind of two degrees of removal.
Anybody that I'd been arrested with and anybody that they
would, had been arrested with counted as people who were
within my network.
So my risk score would be based on my individual history
of arrest and victimization, as well as the histories
of arrest and victimization of people within that
two-degree network of mine.
It was colloquially known as the "heat list."
If you were hot, you were on it.
And they gave you literally a risk score.
At one time, it was zero to 500-plus.
If you're 500-plus, you are a high-risk person.
♪ ♪
And if you made this heat list,
you might find a detective knocking on your front door.
♪ ♪
NARRATOR: Trying to predict
future criminal activity is not a new idea.
Scotland Yard in London began using this approach
by mapping crime events in the 1930s.
But in the 1990s,
it was New York City Police Commissioner William Bratton
who took crime mapping to another level.
BRATTON: I run the New York City Police Department.
My competition is the criminal element.
NARRATOR: Bratton convinced policing agencies across the country
that data-driven policing was the key
to successful policing strategies.
Part of this is to prevent crime in the first place.
♪ ♪
NARRATOR: Bratton was inspired
by the work of his own New York City Transit Police.
As you see all those,
uh, dots on the map,
that's our opponents.
NARRATOR: It was called Charts of the Future,
and credited with cutting subway felonies by 27%
and robberies by a third.
Bratton saw potential.
He ordered all New York City precincts
to systematically map crime,
collect data, find patterns, report back.
The new approach was called CompStat.
BRAYNE: CompStat, I think, in a way, is kind of a precessor
of predictive policing,
in the sense that many of the same principles there--
you know, using data tracking, year-to-dates,
identifying places where law enforcement interventions
could be effective, et cetera--
really laid the groundwork for predictive policing.
♪ ♪
NARRATOR: By the early 2000s,
as computational power increased,
criminologists were convinced this new data trove
could be used in machine learning to create models
that predict when and where
crime would happen in the future.
♪ ♪
REPORTER: L.A. police now say the gunmen opened fire
with a semi-automatic weapon.
NARRATOR: In 2008,
now as chief of the Los Angeles Police Department,
Bratton joined with academics at U.C.L.A.
to help launch a predictive policing system
called PredPol,
powered by a machine learning algorithm.
♪ ♪
ISAAC: PredPol started
as a spin-off of a set of, like,
government contracts that were related to military work.
They were developing
a form of an algorithm that was used to predict I.E.Ds.
(device explodes)
And it was a technique that was used
to also detect aftershocks and seismographic activity.
(dogs barking and whining, objects clattering)
And after those contracts ended,
the company decided they wanted to apply this
in the domain of, of policing
domestically in the United States.
(radio beeping)
NARRATOR: The PredPol model
relies on three types of historical data:
type of crime, crime location, and time of crime,
going back two to five years.
The algorithm is looking for patterns
to identify locations where crime is most likely to occur.
As new crime incidents are reported,
they get folded into the calculation.
The predictions are displayed on a map
as 500 x 500 foot areas that officers are then
directed to patrol.
ISAAC: And then from there, the algorithm says,
"Okay, based on what we know about the kind of
"very recent history,
"where is likely that we'll see crime
in the next day or the next hour?"
♪ ♪
BRAYNE: One of the key reasons
that police start using these tools
is the efficient and even, to a certain extent,
like in their logic,
more fair, um, and, and justifiable allocation
of their police resources.
♪ ♪
NARRATOR: By 2013,
in addition to PredPol, predictive policing systems
developed by companies like HunchLab, IBM, and Palantir
were in use across the country.
(radios running in background)
And computer algorithms
were also being adopted in courtrooms.
BAILIFF: 21CF3810, State of Wisconsin versus Chantille...
KATHERINE FORREST: These tools are used in pretrial determinations,
they're used in sentencing determinations,
and they're used in housing determinations.
They're also used, importantly, in the plea bargaining phase.
They're used really throughout the entire process
to try to do what judges have been doing,
which is the very, very difficult task
of trying to understand and predict
what will a human being do tomorrow, or the next day,
or next month, or three years from now.
ASSISTANT DISTRICT ATTORNEY: Bail forfeited.
He failed to appear 12/13/21.
Didn't even make it to preliminary hearing.
The software tools are an attempt to try to predict
it better than humans can.
MICHELLE HAVAS: On count one, you're charged with
felony intimidation of a victim.
SWEENEY: So, in the United States, you're innocent
until you've been proven guilty, but you've been arrested.
Now that you've been arrested,
a judge has to decide whether or not
you get out on bail,
or how high or low that bail should be.
You're charged with driving on a suspended license.
I've set that bond at $1,000.
No insurance, I've set that bond at $1,000.
ALISON SHAMES: One of the problems is,
judges often are relying on money bond
or financial conditions of release.
JUDGE: So I'm going to lower his fine
to make it a bit more reasonable.
So instead of $250,000 cash,
surety is $100,000.
SHAMES: It allows people who have access to money to be released.
If you are poor, you are often being detained pretrial.
Approximately 70% of the people in jail are there on pretrial.
These are people who are presumed innocent,
but are detained during the pretrial stage of their case.
NARRATOR: Many jurisdictions use pretrial assessment algorithms
with a goal to reduce jail populations and decrease
the impact of judicial bias.
SHAMES: The use of a tool like this takes historical data
and assesses, based on research,
associates factors that are predictive of the two outcomes
that the judge is concerned with.
That's community safety and whether that person
will appear back in court during the pretrial period.
♪ ♪
NARRATOR: Many of these algorithms are based on a concept called
a regression model.
The earliest, called linear regression,
dates back to 19th-century mathematics.
O'NEIL: At the end of the day, machine learning algorithms
do exactly what linear regression does,
which is predict--
based on the initial conditions, the situation they're seeing--
predict what will happen in the future,
whether that's, like, in the next one minute
or the next four years.
NARRATOR: Throughout the United States, over 60 jurisdictions
use predictive algorithms as part of the legal process.
One of the most widely used is COMPAS.
The COMPAS algorithm weighs factors,
including a defendant's answers to a questionnaire,
to provide a risk assessment score.
These scores are used every day by judges to guide decisions
about pretrial detention, bail, and even sentencing.
But the reliability of the COMPAS algorithm
has been questioned.
In 2016, ProPublica published an investigative report
on the COMPAS risk assessment tool.
BENJAMIN: Investigators wanted to see if the scores were accurate
in predicting whether these individuals
would commit a future crime.
And they found two things that were interesting.
One was that the score was remarkably unreliable
in predicting who would commit a, a crime in the future
over this two-year period.
But then the other thing that ProPublica investigators found
was that Black people were much more likely to be deemed
high risk and white people low risk.
NARRATOR: This was true even in cases when the Black person
was arrested for a minor offense and the white person
in question was arrested for a more serious crime.
BENJAMIN: This ProPublica study was one of the first to begin
to burst the bubble of technology
as somehow objective and neutral.
NARRATOR: The article created a national controversy.
But at Dartmouth, a student convinced her professor
they should both be more than stunned.
HANY FARID: As it turns out, one of my students, Julia Dressel,
reads the same article and said,
"This is terrible.
We should do something about it." (chuckles)
This is the difference between an awesome idealistic student
and a jaded, uh, professor.
And I thought, "I think you're right."
And as we were sort of struggling to understand
the underlying roots of the bias in the algorithms,
we asked ourselves a really simple question:
are the algorithms today, are they doing better than humans?
Because presumably, that's why you have these algorithms,
is that they eliminate some of the bias and the prejudices,
either implicit or explicit, in the human judgment.
NARRATOR: To analyze COMPAS's risk assessment accuracy,
they used the crowdsourcing platform Mechanical Turk.
Their online study included 400 participants
who evaluated 1,000 defendants.
FARID: We asked participants to
read a very short paragraph about an actual defendant.
How old they were,
whether they were male or female,
what their prior juvenile conviction record was,
and their prior adult conviction record.
And, importantly, we didn't tell people their race.
And then we ask a very simple question,
"Do you think this person will commit a crime
in the next two years?"
Yes, no.
And again, these are non-experts.
These are people being paid
a couple of bucks online to answer a survey.
No criminal justice experience,
don't know anything about the defendants.
They were as accurate as the commercial software
being used in the courts today,
one particular piece of software.
That was really surprising.
We would've expected a little bit of improvement.
After all, the algorithm has access
to huge amounts of training data.
NARRATOR: And something else puzzled the researchers.
The MTurk workers' answers to questions
about who would commit crimes in the future and who wouldn't
showed a surprising pattern of racial bias,
even though race wasn't indicated
in any of the profiles.
They were more likely to say a person of color
will be high risk when they weren't,
and they were more likely to say that a white person
would not be high risk when in fact they were.
And this made no sense to us at all.
You don't know the race of the person.
How is it possible that you're biased against them?
(radios running in background)
In this country, if you are a person of color,
you are significantly more likely, historically,
to be arrested,
to be charged, and to be convicted of a crime.
So in fact, prior convictions
is a proxy for your race.
Not a perfect proxy, but it is correlated,
because of the historical inequities
in the criminal justice system
and policing in this country.
(siren blaring)
MAN: It's my car, bro, come on, what are y'all doing?
Like, this, this is racial profiling.
NARRATOR: Research indicates a Black person
is five times more likely to be stopped without cause
than a white person.
Black people are at least twice as likely
as white people to be arrested for drug offenses,
even though Black and white people
use drugs at the same rate.
Black people are also about 12 times
more likely to be wrongly convicted of drug crimes.
FORREST: Historically, Black men have been arrested at higher levels
than other populations.
Therefore, the tool predicts that a Black man, for instance,
will be arrested at a rate and recidivate at a rate
that is higher than a white individual.
FARID: And so what was happening is, you know, the big data,
the big machine learning folks are saying,
"Look, we're not giving it race-- it can't be racist."
But that is spectacularly naive,
because we know that other things correlate with race.
In this case, number of prior convictions.
And so when you train an algorithm on historical data,
well, guess what.
It's going to reproduce history-- of course it will.
NARRATOR: Compounding the problem is the fact that
predictive algorithms can't be put on the witness stand
and interrogated about their decision-making processes.
FORREST: Many defendants have had difficulty
getting access to the underlying information
that tells them,
what was the data set that was used to assess me?
What were the inputs that were used?
How were those inputs weighted?
So you've got what can be, these days,
increasingly, a black box.
A lack of transparency.
NARRATOR: Some black box algorithms get their name
from a lack of transparency about the code
and data inputs they use,
which can be deemed proprietary.
But that's not the only kind of black box.
A black box is any system which is so complicated
that you can see what goes in and you can see what comes out,
but it's impossible to understand
what's going on inside it.
All of those steps in the algorithm
are hidden inside phenomenally complex math
and processes.
FARID: And I would argue that when you are using algorithms
in mission-critical applications,
like criminal justice system,
we should not be deploying black box algorithms.
NARRATOR: PredPol, like many predictive platforms,
claimed a proven record for crime reduction.
In 2015, PredPol published its algorithm
in a peer-reviewed journal.
William Isaac and Kristian Lum,
research scientists who investigate
predictive policing platforms,
analyzed the algorithm.
ISAAC: We just kind of saw the algorithm
as going back to the same one or two blocks
every single time.
And that's kind of strange,
because if you had a truly predictive policing system,
you wouldn't necessarily see it going to the same locations
over and over again.
NARRATOR: For their experiment,
Isaac and Lum used a different data set,
public health data, to map illicit drug use in Oakland.
ISAAC: So, a good chunk of the city was kind of
evenly distributed in terms of where
potential illicit drug use might be.
But the police predictions were clustering around areas
where police had, you know,
historically found incidents of illicit drug use.
Specifically, we saw significant numbers of neighborhoods
that were predominantly non-white and lower-income
being deliberate targets of the predictions.
NARRATOR: Even though illicit drug use was a citywide problem,
the algorithm focused its predictions
on low-income neighborhoods and communities of color.
ISAAC: The reason why is actually really important.
It's very hard to divorce
these predictions from those histories
and legacies of over-policing.
As a result of that, they manifest themselves in the data.
NARRATOR: In an area where there is more police presence,
more crime is uncovered.
The crime data indicates to the algorithm
that the heavily policed neighborhood
is where future crime will be found,
even though there may be other neighborhoods
where crimes are being committed at the same or higher rate.
ISAAC: Every new prediction that you generate
is going to be increasingly dependent
on the behavior of the algorithm in the past.
So, you know, if you go ten days, 20 days,
30 days into the future, right, after using an algorithm,
all of those predictions have changed the behavior
of the police department
and are now being folded back into the next day's prediction.
NARRATOR: The result can be a feedback loop
that reinforces historical policing practices.
SWEENEY: All of these different types
of machine learning algorithms are all trying to help us
figure out, are there some patterns in this data?
It's up to us to then figure out,
are those legitimate patterns, do they,
are they useful patterns?
Because the computer has no idea.
It didn't make a logical association.
It just made it, made a correlation.
MING: My favorite definition of artificial intelligence
is, it's any autonomous system
that can make decisions under uncertainty.
You can't make decisions under uncertainty without bias.
In fact, it's impossible to escape from having bias.
It's a mathematical reality
about any intelligent system, even us.
(siren blaring in distance)
NARRATOR: And even if the goal is to get rid of prejudice,
bias in the historical data can undermine that objective.
Amazon discovered this
when they began a search for top talent
with a hiring algorithm whose training data
depended on hiring successes from the past.
MING: Amazon, somewhat famously within the A.I. industry,
they tried to build a hiring algorithm.
They had a massive data set.
They had all the right answers,
because they knew literally who got hired
and who got that promotion in their first year.
(typing)
NARRATOR: The company created multiple models
to review past candidates' risumis
and identify some 50,000 key terms.
MING: What Amazon actually wanted to achieve
was to diversify their hiring.
Amazon, just like every other tech company,
and a lot of other companies, as well,
has enormous bias built into its hiring history.
It was always biased, strongly biased, in favor of men,
in favor, generally,
of white or sometimes Asian men.
Well, they went and built a hiring algorithm.
And sure enough, this thing was
the most sexist recruiter you could imagine.
If you said the word "women's" in your risumi,
then it wouldn't hire you.
If you went to a women's college,
it didn't want to hire you.
So they take out all the gender markers,
and all of the women's colleges--
all the things that explicitly says,
"This is a man," and, "This is a woman,"
or even the ones that, obviously, implicitly say it.
So they did that.
And then they trained up their new deep neural network
to decide who Amazon would hire.
And it did something amazing.
It did something no human could do.
It figured out who was a woman and it wouldn't hire them.
It was able to look through
all of the correlations that existed
in that massive data set
and figure out which ones most strongly correlated
with someone getting a promotion.
And the single biggest correlate
of getting a promotion was being a man.
And it figured those patterns out and didn't hire women.
NARRATOR: Amazon abandoned its hiring algorithm in 2017.
Remember the way machine learning works, right?
It's like a student who doesn't really understand
the material in the class.
They got a bunch of questions, they got a bunch of answers.
And now they're trying to pattern match
for a new question and say, "Oh, wait.
"Let me find an answer that looks pretty much
like the questions and answers I saw before."
The algorithm only worked because someone has said,
"Oh, this person whose data you have,
"they were a good employee.
This other person was a bad employee," or, "This person
performed well," or, "This person did not perform well."
O'NEIL: Because algorithms don't just look for patterns,
they look for patterns of success, however it's defined.
But the definition of success is really critically important
to what that end up, ends up being.
And a lot of, a lot of opinion
is embedded in, what, what does success look like?
NARRATOR: In the case of algorithms,
human choices play a critical role.
O'NEIL: The data itself was curated.
Someone decided what data to collect.
Somebody decided what data was not relevant, right?
And they don't exclude it necessarily
intentionally-- they could be blind spots.
NARRATOR: The need to identify such oversights
becomes more urgent
as technology takes on more decision making.
♪ ♪
Consider facial recognition technology,
used by law enforcement
in cities around the world for surveillance.
In Detroit, 2018,
law enforcement looked to facial recognition technology
when $3,800 worth of watches
were stolen from an upscale boutique.
Police ran a still frame from the shop's surveillance video
through their facial recognition system to find a match.
How do I turn a face into numbers
that equations can act with?
You turn the individual pixels in the picture of that face
into values.
What it's really looking for are complex patterns
across those pixels.
The sequence of taking a pattern of numbers
and transforming it
into little edges and angles,
then transforming that into eyes and cheekbones and mustaches.
NARRATOR: To find that match,
the system can be trained on billions of photographs.
Facial recognition uses a class of machine learning
called deep learning.
The models built by deep learning techniques
are called neural networks.
VENKATASUBRAMANIAN: A neural network
is, you know, stylized as, you know, trying to model
how neural pathways work in the brain.
You can think of a neural network
as a collection of neurons.
So you put some values into a neuron,
and if they're, sufficiently, they add up to some number,
or they cross some threshold,
this one will fire and send off a new number to the next neuron.
NARRATOR: At a certain threshold,
the neuron will fire to the next neuron.
If it's below the threshold, the neuron doesn't fire.
This process repeats and repeats
across hundreds, possibly thousands of layers,
making connections like the neurons in our brain.
♪ ♪
The output is a predictive match.
Based on a facial recognition match,
in January 2020, the police arrested Robert Williams
for the theft of the watches.
The next day, he was released.
Not only did Williams have an alibi,
but it wasn't his face.
MING: To be very blunt about it, these algorithms are probably
dramatically over-trained on white faces.
♪ ♪
So, of course, algorithms that start out bad
can be improved, in general.
The Gender Shades project found that
certain facial recognition technology,
when they actually tested it on Black women,
it was 65% accurate, whereas for white men, it was 99% accurate.
How did they improve it? Because they did.
They built an algorithm
that was trained on more diverse data.
So I don't think it's completely a lost cause
to improve algorithms to be better.
MAN (in ad voiceover): I used to think my job was all about arrests.
NEDY: LESLIE KEN There was a commercial a few years ago
that showed a police officer going to a gas station
and then waiting for the criminal to show up.
MAN: We analyze crime data,
spot patterns,
and figure out where to send patrols.
They said, "Well, our algorithm will tell you exactly
where the crime, the next crime is going to take place."
Well, that's just silly, uh, it, that's not how it works.
MAN: By stopping it before it happens.
(sighs)
MAN: Let's build a smarter planet.
♪ ♪
JOEL CAPLAN: Understanding what it is about these places
that enable crime problems
to emerge and/or persist.
NARRATOR: At Rutgers University,
the researchers who invented
the crime mapping platform called Risk Terrain Modeling,
or RTM,
bristle at the term "predictive policing."
CAPLAN (voiceover): We don't want to predict, we want to prevent.
I worked as a police officer a long time ago,
in the early 2000s.
Police collected data for as long as police have existed.
Now there's a greater recognition
that data can have value.
But it's not just about the data.
It's about how you analyze it, how you use those results.
There's only two data sets that risk terrain modeling uses.
These data sets are local,
current information about crime incidents within a given area
and information about environmental features
that exist in that landscape,
such as bars, fast food restaurants,
convenience stores, schools, parks, alleyways.
KENNEDY: The algorithm is basically
the relationship between these environmental features
and the, the outcome data, which in this case is crime.
The algorithm provides you with a map
of the distribution of the risk values.
ALEJANDRO GIME NEZ-SANTANA: This is the highest-risk area,
on this commercial corridor on Bloomfield Avenue.
NARRATOR: But the algorithm isn't intended for use just by police.
Criminologist Alejandro Giminez-Santana
leads the Newark Public Safety Collaborative,
a collection of 40 community organizations.
They use RTM as a diagnostic tool
to understand not just where crime may happen next,
but why.
GIME NEZ-SANTANA: Through RTM, we identify this commercial corridor
on Bloomfield Avenue, which is where we are right now,
as a risky area for auto theft due to car idling.
So why is this space
particularly problematic when it comes to auto theft?
One is because we're in a commercial corridor,
where there's high density of people
who go to the beauty salon or to go to a restaurant.
Uber delivery and Uber Eats,
delivery people who come to grab orders that also,
and leave their cars running
create the conditions for this crime
to be concentrated in this particular area.
What the data showed us was,
there was a tremendous rise in auto vehicle thefts.
But we convinced the police department
to take a more social service approach.
NARRATOR: Community organizers convinced police
not to ticket idling cars,
and let organizers create
an effective public awareness poster campaign instead.
And we put it out to the Newark students
to submit in this flyer campaign,
and have their artwork on the actual flyer.
GIME NEZ-SANTANA: As you can see, this is the commercial corridor
on Bloomfield Avenue.
The site score shows a six,
which means that we are at the highest risk of auto theft
in this particular location.
And as I move closer to the end
of the commercial corridor, the site risk score is coming down.
NARRATOR: This is the first time in Newark
that police data for crime occurrences
have been shared widely with community members.
ELVIS PEREZ: The kind of data we share is incident-related data--
sort of time, location, that sort of information.
We don't discuss any private arrest information.
We're trying to avoid a crime.
NARRATOR: In 2019,
Caplan and Kennedy formed a start-up at Rutgers
to meet the rising demand for their technology.
Despite the many possible applications for RTM,
from tracking public health issues
to understanding vehicle crashes,
law enforcement continues to be its principal application.
Like any other technology,
risk terrain modeling can be used for the public good
when people use it wisely.
♪ ♪
We as academics and scientists, we actually need to be critical,
because it could be the best model in the world,
it could be very good predictions,
but how you use those predictions
matters, in some ways, even more.
REPORTER: The police department
had revised the SSL numerous times...
NARRATOR: In 2019,
Chicago's inspector general contracted the RAND Corporation
to evaluate the Strategic Subject List,
the predictive policing platform
that incorporated Papachristos's research on social networks.
PAPACHRISTOS: I never wanted to go down this path
of who was the person that was the potential suspect.
And that problem is not necessarily
with the statistical model, it's the fact that someone
took victim and made him an offender.
You've criminalized someone who is at risk,
that you should be prioritizing saving their life.
NARRATOR: It turned out that some 400,000 people were included on the SSL.
Of those, 77% were Black or Hispanic.
The inspector general's audit revealed
that SSL scores were unreliable.
The Rand Corporation found the program had no impact
on homicide or victimization rates.
(protesters chanting)
NARRATOR: The program was shut down.
But data collection continues
to be essential to law enforcement.
♪ ♪
O'NEIL: There are things about us that we might not even
be aware of that are sort of being collected
by the data brokers
and will be held against us for the rest of our lives--
held against people forever, digitally.
NARRATOR: Data is produced and collected.
But is it accurate?
And can the data be properly vetted?
PAPACHRISTOS: And that was one of the critiques
of not just the Strategic Subjects List,
but the gang database in Chicago.
Any data source that treats data as a stagnant, forever condition
is a problem.
WOMAN: The gang database has been around for four years.
It'll be five in January.
We want to get rid of surveillance
in Black and brown communities.
BENJAMIN: In places like Chicago,
in places like L.A., where I grew up,
there are gang databases with tens of thousands
of people listed, their names listed in these databases.
Just by simply having a certain name
and coming from a certain ZIP code
could land you in these databases.
Do you all feel safe in Chicago?
DARRELL DACRES: The cops pulled up out of nowhere.
Didn't ask any questions, just immediately start beating on us.
And basically were saying, like,
what are, what are we doing over here, you know, like,
in this, in this gangbang area?
I was already labeled as a gangbanger
from that area because of where I lived.
I, I just happened to live there.
NARRATOR: The Chicago gang database
is shared with hundreds of law enforcement agencies.
Even if someone is wrongly included,
there is no mechanism to have their name removed.
If you try to apply for an apartment,
or if you try to apply for a job or a college,
or even in a, um, a house, it will show
that you are in this record of a gang database.
I was arrested for peacefully protesting.
And they told me that, "Well,
you're in the gang database."
But I was never in no gang.
MAN: Because you have a gang designation,
you're a security threat group,
right?
NARRATOR: Researchers and activists
have been instrumental in dismantling
some of these systems.
And so we continue to push back.
I mean, the fight is not going to finish
until we get rid of the database.
♪ ♪
FERGUSON: I think what we're seeing now
is not a move away from data.
It's just a move away from this term "predictive policing."
But we're seeing big companies,
big tech, enter the policing space.
We're seeing the reality that almost all policing now
is data-driven.
You're seeing these same police departments
invest heavily in the technology,
including other forms of surveillance technology,
including other forms of databases
to sort of manage policing.
(chanting): We want you out!
NARRATOR: More citizens are calling for regulations
to audit algorithms
and guarantee they're accomplishing what they promise
without harm.
BRAYNE: Ironically, there is very little data
on police use of big data.
And there is no systematic data
at a national level on how these tools are used.
The deployment of these tools
so far outpaces
legal and regulatory responses to them.
What you have happening
is essentially this regulatory Wild West.
O'NEIL: And we're, like, "Well, it's an algorithm,
let's, let's just throw it into production."
Without testing it to whether
it "works" sufficiently, um, at all.
NARRATOR: Multiple requests for comment
from police agencies and law enforcement officials
in several cities, including Chicago and New York,
were either declined or went unanswered.
♪ ♪
Artificial intelligence must serve people,
and therefore artificial intelligence
must always comply with people's rights.
NARRATOR: The European Union is preparing to implement legislation
to regulate artificial intelligence.
In 2021, bills to regulate data science algorithms
were introduced in 17 states,
and enacted in Alabama, Colorado,
Illinois, and Mississippi.
SWEENEY: If you look carefully on electrical devices,
you'll see "U.L.," for Underwriters Laboratory.
That's a process that came about
so that things, when you plugged them in,
didn't blow up in your hand.
That's the same kind of idea
that we need in these algorithms.
O'NEIL: We can adjust it to make it better than the past,
and we can do it carefully,
and we can do it with, with precision
in an ongoing conversation about what it means to us
that it is, it's biased in the right way.
I don't think you remove bias,
but you get to a bias that you can live with,
that you, you think is moral.
To be clear, like, I, I think we can do better,
but often doing better
would look like we don't use this at all.
(radio running)
FARID: There's nothing fundamentally wrong
with trying to predict the future,
as long as you understand how the algorithms are working,
how are they being deployed.
What is the consequence of getting it right?
And most importantly is,
what is the consequence of getting it wrong?
OFFICER: Keep your hands on the steering wheel!
MAN: My hands haven't moved off the steering wheel!
MAN 2: Are you gonna arrest me?
MAN 1: Officer, what are we here for?
OFFICER: We just want to talk with...
♪ ♪
ANNOUNCER: This program is available with PBS Passport
and on Amazon Prime Video.
♪ ♪