Facebook's Secret Censorship Rules Protect White Men but Not Black Children
From Hate Speech
Friday, June 30, 2017
By Julia Angwin and Hannes Grassegger, ProPublica | Report
Facebook CEO Mark Zuckerberg speaks at the Facebook Developer Conference in
San Jose, California, April 18, 2017. (Photo: Anthony Quintano)
In the wake of a terrorist attack in London earlier this month, a US
congressman wrote a Facebook post in which he called for the slaughter of
"radicalized" Muslims. "Hunt them, identify them, and kill them," declared
US Rep. Clay Higgins, a Louisiana Republican. "Kill them all. For the sake
of all that is good and righteous. Kill them all."
Higgins' plea for violent revenge went untouched by Facebook workers who
scour the social network deleting offensive speech.
But a May posting on Facebook by Boston poet and Black Lives Matter activist
Didi Delgado drew a different response.
"All white people are racist. Start from this reference point, or you've
already failed," Delgado wrote. The post was removed and her Facebook
account was disabled for seven days.
A trove of internal documents reviewed by ProPublica sheds new light on the
secret guidelines that Facebook's censors use to distinguish between hate
speech and legitimate political expression. The documents reveal the
rationale behind seemingly inconsistent decisions. For instance, Higgins'
incitement to violence passed muster because it targeted a specific
sub-group of Muslims -- those that are "radicalized" -- while Delgado's post
was deleted for attacking whites in general.
Over the past decade, the company has developed hundreds of rules, drawing
elaborate distinctions between what should and shouldn't be allowed, in an
effort to make the site a safe place for its nearly 2 billion users. The
issue of how Facebook monitors this content has become increasingly
prominent in recent months, with the rise of "fake news" -- fabricated
stories that circulated on Facebook like "Pope Francis Shocks the World,
Endorses Donald Trump For President, Releases Statement" -- and growing
concern that terrorists are using social media for recruitment.
While Facebook was credited during the 2010-2011 "Arab Spring" with
facilitating uprisings against authoritarian regimes, the documents suggest
that, at least in some instances, the company's hate-speech rules tend to
favor elites and governments over grassroots activists and racial
minorities. In so doing, they serve the business interests of the global
company, which relies on national governments not to block its service to
their citizens.
One Facebook rule, which is cited in the documents but that the company said
is no longer in effect, banned posts that praise the use of "violence to
resist occupation of an internationally recognized state." The company's
workforce of human censors, known as content reviewers, has deleted posts by
activists and journalists in disputed territories such as Palestine,
Kashmir, Crimea and Western Sahara.
One document trains content reviewers on how to apply the company's global
hate speech algorithm. The slide identifies three groups: female drivers,
black children and white men. It asks: Which group is protected from hate
speech? The correct answer: white men.
The reason is that Facebook deletes curses, slurs, calls for violence and
several other types of attacks only when they are directed at "protected
categories" -- based on race, sex, gender identity, religious affiliation,
national origin, ethnicity, sexual orientation and serious
disability/disease. It gives users broader latitude when they write about
"subsets" of protected categories. White men are considered a group because
both traits are protected, while female drivers and black children, like
radicalized Muslims, are subsets, because one of their characteristics is
not protected.
Behind this seemingly arcane distinction lies a broader philosophy. Unlike
American law, which permits preferences such as affirmative action for
racial minorities and women for the sake of diversity or redressing
discrimination, Facebook's algorithm is designed to defend all races and
genders equally.
"Sadly," the rules are "incorporating this color-blindness idea which is not
in the spirit of why we have equal protection," said Danielle Citron, a law
professor and expert on information privacy at the University of Maryland.
This approach, she added, will "protect the people who least need it and
take it away from those who really need it."
But Facebook says its goal is different -- to apply consistent standards
worldwide. "The policies do not always lead to perfect outcomes," said
Monika Bickert, head of global policy management at Facebook. "That is the
reality of having policies that apply to a global community where people
around the world are going to have very different ideas about what is OK to
share."
Facebook's rules constitute a legal world of their own. They stand in sharp
contrast to the United States' First Amendment protections of free speech,
which courts have interpreted to allow exactly the sort of speech and
writing censored by the company's hate speech algorithm. But they also
differ -- for example, in permitting postings that deny the Holocaust --
from more restrictive European standards.
The company has long had programs to remove obviously offensive material
like child pornography from its stream of images and commentary. Recent
articles in the Guardian and Süddeutsche Zeitung have detailed the difficult
choices that Facebook faces regarding whether to delete posts containing
graphic violence, child abuse, revenge porn and self-mutilation.
The challenge of policing political expression is even more complex. The
documents reviewed by ProPublica indicate, for example, that Donald Trump's
posts about his campaign proposal to ban Muslim immigration to the United
States violated the company's written policies against "calls for exclusion"
of a protected group. As The Wall Street Journal reported last year,
Facebook exempted Trump's statements from its policies at the order of Mark
Zuckerberg, the company's founder and chief executive.
The company recently pledged to nearly double its army of censors to 7,500,
up from 4,500, in response to criticism of a video posting of a murder.
Their work amounts to what may well be the most far-reaching global
censorship operation in history. It is also the least accountable: Facebook
does not publish the rules it uses to determine what content to allow and
what to delete.
Users whose posts are removed are not usually told what rule they have
broken, and they cannot generally appeal Facebook's decision. Appeals are
currently only available to people whose profile, group or page is removed.
The company has begun exploring adding an appeals process for people who
have individual pieces of content deleted, according to Bickert. "I'll be
the first to say that we're not perfect every time," she said.
***
Facebook is not required by US law to censor content. A 1996 federal law
gave most tech companies, including Facebook, legal immunity for the content
users post on their services. The law, section 230 of the Telecommunications
Act, was passed after Prodigy was sued and held liable for defamation for a
post written by a user on a computer message board.
The law freed up online publishers to host online forums without having to
legally vet each piece of content before posting it, the way that a news
outlet would evaluate an article before publishing it. But early tech
companies soon realized that they still needed to supervise their chat rooms
to prevent bullying and abuse that could drive away users.
America Online convinced thousands of volunteers to police its chat rooms in
exchange for free access to its service. But as more of the world connected
to the internet, the job of policing became more difficult and companies
started hiring workers to focus on it exclusively. Thus the job of content
moderator -- now often called content reviewer -- was born.
In 2004, attorney Nicole Wong joined Google and persuaded the company to
hire its first-ever team of reviewers, who responded to complaints and
reported to the legal department. Google needed "a rational set of policies
and people who were trained to handle requests," for its online forum called
Groups, she said.
Google's purchase of YouTube in 2006 made deciding what content was
appropriate even more urgent. "Because it was visual, it was universal,"
Wong said.
While Google wanted to be as permissive as possible, she said, it soon had
to contend with controversies such as a video mocking the King of Thailand,
which violated Thailand's laws against insulting the king. Wong visited
Thailand and was impressed by the nation's reverence for its monarch, so she
reluctantly agreed to block the video -- but only for computers located in
Thailand.
Since then, selectively banning content by geography -- called
"geo-blocking" -- has become a more common request from governments. "I
don't love traveling this road of geo-blocking," Wong said, but "it's ended
up being a decision that allows companies like Google to operate in a lot of
different places."
For social networks like Facebook, however, geo-blocking is difficult
because of the way posts are shared with friends across national boundaries.
If Facebook geo-blocks a user's post, it would only appear in the news feeds
of friends who live in countries where the geo-blocking prohibition doesn't
apply. That can make international conversations frustrating, with bits of
the exchange hidden from some participants.
As a result, Facebook has long tried to avoid using geography-specific rules
when possible, according to people familiar with the company's thinking.
However, it does geo-block in some instances, such as when it complied with
a request from France to restrict access within its borders to a photo taken
after the Nov. 13, 2015, terrorist attack at the Bataclan concert hall in
Paris.
Bickert said Facebook takes into consideration the laws in countries where
it operates, but doesn't always remove content at a government's request.
"If there is something that violates a country's law but does not violate
our standards," Bickert said, "we look at who is making that request: Is it
the appropriate authority? Then we check to see if it actually violates the
law. Sometimes we will make that content unavailable in that country only."
Facebook's goal is to create global rules. "We want to make sure that people
are able to communicate in a borderless way," Bickert said.
***
Founded in 2004, Facebook began as a social network for college students. As
it spread beyond campus, Facebook began to use content moderation as a way
to compete with the other leading social network of that era, MySpace.
MySpace had positioned itself as the nightclub of the social networking
world, offering profile pages that users could decorate with online glitter,
colorful layouts and streaming music. It didn't require members to provide
their real names and was home to plenty of nude and scantily clad
photographs. And it was being investigated by law-enforcement agents across
the country who worried it was being used by sexual predators to prey on
children. (In a settlement with 49 state attorneys general, MySpace later
agreed to strengthen protections for younger users.)
By comparison, Facebook was the buttoned-down Ivy League social network --
all cool grays and blues. Real names and university affiliations were
required. Chris Kelly, who joined Facebook in 2005 and was its first general
counsel, said he wanted to make sure Facebook didn't end up in law
enforcement's crosshairs, like MySpace.
"We were really aggressive about saying we are a no-nudity platform," he
said.
The company also began to tackle hate speech. "We drew some difficult lines
while I was there -- Holocaust denial being the most prominent," Kelly said.
After an internal debate, the company decided to allow Holocaust denials but
reaffirmed its ban on group-based bias, which included anti-Semitism. Since
Holocaust denial and anti-Semitism frequently went together, he said, the
perpetrators were often suspended regardless.
"I've always been a pragmatist on this stuff," said Kelly, who left Facebook
in 2010. "Even if you take the most extreme First Amendment positions, there
are still limits on speech."
By 2008, the company had begun expanding internationally but its censorship
rulebook was still just a single page with a list of material to be excised,
such as images of nudity and Hitler. "At the bottom of the page it said,
'Take down anything else that makes you feel uncomfortable,'" said Dave
Willner, who joined Facebook's content team that year.
Willner, who reviewed about 15,000 photos a day, soon found the rules were
not rigorous enough. He and some colleagues worked to develop a coherent
philosophy underpinning the rules, while refining the rules themselves. Soon
he was promoted to head the content policy team.
By the time he left Facebook in 2013, Willner had shepherded a 15,000-word
rulebook that remains the basis for many of Facebook's content standards
today.
"There is no path that makes people happy," Willner said. "All the rules are
mildly upsetting." Because of the volume of decisions -- many millions per
day -- the approach is "more utilitarian than we are used to in our justice
system," he said. "It's fundamentally not rights-oriented."
Willner's then-boss, Jud Hoffman, who has since left Facebook, said that the
rules were based on Facebook's mission of "making the world more open and
connected." Openness implies a bias toward allowing people to write or post
what they want, he said.
But Hoffman said the team also relied on the principle of harm articulated
by John Stuart Mill, a 19th-century English political philosopher. It states
"that the only purpose for which power can be rightfully exercised over any
member of a civilized community, against his will, is to prevent harm to
others." That led to the development of Facebook's "credible threat"
standard, which bans posts that describe specific actions that could
threaten others, but allows threats that are not likely to be carried out.
Eventually, however, Hoffman said "we found that limiting it to physical
harm wasn't sufficient, so we started exploring how free expression
societies deal with this."
The rules developed considerable nuance. There is a ban against pictures of
Pepe the Frog, a cartoon character often used by "alt-right" white
supremacists to perpetrate racist memes, but swastikas are allowed under a
rule that permits the "display [of] hate symbols for political messaging."
In the documents examined by ProPublica, which are used to train content
reviewers, this rule is illustrated with a picture of Facebook founder Mark
Zuckerberg that has been manipulated to apply a swastika to his sleeve.
The documents state that Facebook relies, in part, on the US State
Department's list of designated terrorist organizations, which includes
groups such as al-Qaida, the Taliban and Boko Haram. But not all groups
deemed terrorist by one country or another are included: A recent
investigation by the Pakistan newspaper Dawn found that 41 of the 64
terrorist groups banned in Pakistan were operational on Facebook.
There is also a secret list, referred to but not included in the documents,
of groups designated as hate organizations that are banned from Facebook.
That list apparently doesn't include many Holocaust denial and white
supremacist sites that are up on Facebook to this day, such as a group
called "Alt-Reich Nation." A member of that group was recently charged with
murdering a black college student in Maryland.
As the rules have multiplied, so have exceptions to them. Facebook's
decision not to protect subsets of protected groups arose because some
subgroups such as "female drivers" didn't seem especially sensitive. The
default position was to allow free speech, according to a person familiar
with the decision-making.
After the wave of Syrian immigrants began arriving in Europe, Facebook added
a special "quasi-protected" category for migrants, according to the
documents. They are only protected against calls for violence and
dehumanizing generalizations, but not against calls for exclusion and
degrading generalizations that are not dehumanizing. So, according to one
document, migrants can be referred to as "filthy" but not called "filth."
They cannot be likened to filth or disease "when the comparison is in the
noun form," the document explains.
Facebook also added an exception to its ban against advocating for anyone to
be sent to a concentration camp. "Nazis should be sent to a concentration
camp," is allowed, the documents state, because Nazis themselves are a hate
group.
***
The rule against posts that support violent resistance against a foreign
occupier was developed because "we didn't want to be in a position of
deciding who is a freedom fighter," Willner said. Facebook has since dropped
the provision and revised its definition of terrorism to include
nongovernmental organizations that carry out premeditated violence "to
achieve a political, religious or ideological aim," according to a person
familiar with the rules.
The Facebook policy appears to have had repercussions in many of the at
least two dozen disputed territories around the world. When Russia occupied
Crimea in March 2014, many Ukrainians experienced a surge in Facebook
banning posts and suspending profiles. Facebook's director of policy for the
region, Thomas Myrup Kristensen, acknowledged at the time that it "found a
small number of accounts where we had incorrectly removed content. In each
case, this was due to language that appeared to be hate speech but was being
used in an ironic way. In these cases, we have restored the content."
Katerina Zolotareva, 34, a Kiev-based Ukrainian working in communications,
has been blocked so often that she runs four accounts under her name.
Although she supported the "Euromaidan" protests in February 2014 that
antagonized Russia, spurring its military intervention in Crimea, she
doesn't believe that Facebook took sides in the conflict. "There is war in
almost every field of Ukrainian life," she says, "and when war starts, it
also starts on Facebook."
In Western Sahara, a disputed territory occupied by Morocco, a group of
journalists called Equipe Media say their account was disabled by Facebook,
their primary way to reach the outside world. They had to open a new
account, which remains active.
"We feel we have never posted anything against any law," said Mohammed
Mayarah, the group's general coordinator. "We are a group of media
activists. We have the aim to break the Moroccan media blockade imposed
since it invaded and occupied Western Sahara."
In Israel, which captured territory from its neighbors in a 1967 war and has
occupied it since, Palestinian groups are blocked so often that they have
their own hashtag, #FbCensorsPalestine, for it. Last year, for instance,
Facebook blocked the accounts of several editors for two leading Palestinian
media outlets from the West Bank -- Quds News Network and Sheebab News
Agency. After a couple of days, Facebook apologized and un-blocked the
journalists' accounts. Earlier this year, Facebook blocked the account of
Fatah, the Palestinian Authority's ruling party -- then un-blocked it and
apologized.
Last year India cracked down on protesters in Kashmir, shooting pellet guns
at them and shutting off cellphone service. Local insurgents are seeking
autonomy for Kashmir, which is also caught in a territorial tussle between
India and Pakistan. Posts of Kashmir activists were being deleted, and
members of a group called the Kashmir Solidarity Network found that all of
their Facebook accounts had been blocked on the same day.
Ather Zia, a member of the network and a professor of anthropology at the
University of Northern Colorado, said that Facebook restored her account
without explanation after two weeks. "We do not trust Facebook any more,"
she said. "I use Facebook, but it's almost this idea that we will be able to
create awareness but then we might not be on it for long."
***
The rules are one thing. How they're applied is another. Bickert said
Facebook conducts weekly audits of every single content reviewer's work to
ensure that its rules are being followed consistently. But critics say that
reviewers, who have to decide on each post within seconds, may vary in both
interpretation and vigilance.
Facebook users who don't mince words in criticizing racism and police
killings of racial minorities say that their posts are often taken down. Two
years ago, Stacey Patton, a journalism professor at historically black
Morgan State University in Baltimore, posed a provocative question on her
Facebook page. She asked why "it's not a crime when White freelance
vigilantes and agents of 'the state' are serial killers of unarmed Black
people, but when Black people kill each other then we are 'animals' or
'criminals.'"
Although it doesn't appear to violate Facebook's policies against hate
speech, her post was immediately removed, and her account was disabled for
three days. Facebook didn't tell her why. "My posts get deleted about once a
month," said Patton, who often writes about racial issues. She said she also
is frequently put in Facebook "jail" -- locked out of her account for a
period of time after a posting that breaks the rules.
"It's such emotional violence," Patton said. "Particularly as a black
person, we're always having these discussions about mass incarceration, and
then here's this fiber-optic space where you can express yourself. Then you
say something that some anonymous person doesn't like and then you're in
'jail.'"
Didi Delgado, whose post stating that "white people are racist" was deleted,
has been banned from Facebook so often that she has set up an account on
another service called Patreon, where she posts the content that Facebook
suppressed. In May, she deplored the increasingly common Facebook censorship
of black activists in an article for Medium titled "Mark Zuckerberg Hates
Black People."
Facebook also locked out Leslie Mac, a Michigan resident who runs a service
called SafetyPinBox where subscribers contribute financially to "the fight
for black liberation," according to her site. Her offense was writing a post
stating "White folks. When racism happens in public -- YOUR SILENCE IS
VIOLENCE."
The post does not appear to violate Facebook's policies. Facebook apologized
and restored her account after TechCrunch wrote an article about Mac's
punishment. Since then, Mac has written many other outspoken posts. But, "I
have not had a single peep from Facebook," she said, while "not a single one
of my black female friends who write about race or social justice have not
been banned."
"My takeaway from the whole thing is: If you get publicity, they clean it
right up," Mac said. Even so, like most of her friends, she maintains a
separate Facebook account in case her main account gets blocked again.
Negative publicity has spurred other Facebook turnabouts as well. Consider
the example of the iconic news photograph of a young naked girl running from
a napalm bomb during the Vietnam War. Kate Klonick, a Ph.D. candidate at
Yale Law School who has spent two years studying censorship operations at
tech companies, said the photo had likely been deleted by Facebook thousands
of times for violating its ban on nudity.
But last year, Facebook reversed itself after Norway's leading newspaper
published a front-page open letter to Zuckerberg accusing him of "abusing
his power" by deleting the photo from the newspaper's Facebook account.
Klonick said that while she admires Facebook's dedication to policing
content on its website, she fears it is evolving into a place where
celebrities, world leaders and other important people "are
disproportionately the people who have the power to update the rules."
***
In December 2015, a month after terrorist attacks in Paris killed 130
people, the European Union began pressuring tech companies to work harder to
prevent the spread of violent extremism online.
After a year of negotiations, Facebook, Microsoft, Twitter and YouTube
agreed to the European Union's hate speech code of conduct, which commits
them to review and remove the majority of valid complaints about illegal
content within 24 hours and to be audited by European regulators. The first
audit, in December, found that the companies were only reviewing 40 percent
of hate speech within 24 hours, and only removing 28 percent of it. Since
then, the tech companies have shortened their response times to reports of
hate speech and increased the amount of content they are deleting, prompting
criticism from free-speech advocates that too much is being censored.
Now the German government is considering legislation that would allow social
networks such as Facebook to be fined up to 50 million euros if they don't
remove hate speech and fake news quickly enough. Facebook recently posted an
article assuring German lawmakers that it is deleting about 15,000 hate
speech posts a month. Worldwide, over the last two months, Facebook deleted
about 66,000 hate speech posts per week, vice president Richard Allan said
in a statement Tuesday on the company's site.
Among posts that Facebook didn't delete were Donald Trump's comments on
Muslims. Days after the Paris attacks, Trump, then running for president,
posted on Facebook "calling for a total and complete shutdown of Muslims
entering the United States until our country's representatives can figure
out what is going on."
Candidate Trump's posting -- which has come back to haunt him in court
decisions voiding his proposed travel ban -- appeared to violate Facebook's
rules against "calls for exclusion" of a protected religious group.
Zuckerberg decided to allow it because it was part of the political
discourse, according to people familiar with the situation.
However, one person close to Facebook's decision-making said Trump may also
have benefited from the exception for sub-groups. A Muslim ban could be
interpreted as being directed against a sub-group, Muslim immigrants, and
thus might not qualify as hate speech against a protected category.
This piece was reprinted by Truthout with permission or license. It may
not be reproduced in any form without permission or license from the source.
Hannes Grassegger
Hannes Grassegger is a reporter for Das Magazin and Reportagen Magazine
based in Zurich.
Julia Angwin
Julia Angwin is a senior reporter at ProPublica. From 2000 to 2013, she was
a reporter at The Wall Street Journal, where she led a privacy investigative
team that was a finalist for a Pulitzer Prize in explanatory reporting in
2011 and won a Gerald Loeb Award in 2010.
Related Stories
Korryn Gaines Blackout Signals Expansion of Law Enforcement's Special
Relationship With Facebook
By Kenneth Lipp, The District Sentinel | News Analysis
Facebook Doesn't Tell Users Everything It Really Knows About Them
By Julia Angwin, Surya Mattu, Terry Parris Jr., ProPublica | Report
Police Are Using Facebook and Apple to Data-Mine Accounts of Trump Protest
Arrestees
By Sarah Lazare, AlterNet | Report
Show Comments