View on GitHub

hatespeechdata

Catalog of abusive language data (PLoS 2020)

Hate Speech Dataset Catalogue

This page catalogues datasets annotated for hate speech, online abuse, and offensive language. They may be useful for e.g. training a natural language processing system to detect this language.

The list is maintained by Leon Derczynski, Bertie Vidgen, Hannah Rose Kirk, Pica Johansson, Yi-Ling Chung, Mads Guldborg Kjeldgaard Kongsbak, Laila Sprejer, and Philine Zeinert.

We provide a list of datasets and keywords. If you would like to contribute to our catalogue or add your dataset, please see the instructions for contributing.

If you use these resources, please cite (and read!) our paper: Directions in Abusive Language Training Data: Garbage In, Garbage Out. And if you would like to find other resources for researching online hate, visit The Alan Turing Institute’s Online Hate Research Hub or read The Alan Turing Institute’s Reading List on Online Hate and Abuse Research.

If you’re looking for a good paper on online hate training datasets (beyond our paper, of course!) then have a look at ‘Resources and benchmark corpora for hate speech detection: a systematic review’ by Poletto et al. in Language Resources and Evaluation.

Please send contributions via github pull request. You can do this by visiting the source code on github and clicking the edit icon (a pencil, above the text, on the right) - more details below. There’s a commented-out markdown template at the top of this file. Accompanying data statements preferred for all corpora.

Datasets Table of Contents

List of datasets

Albanian

Detecting Abusive Albanian

Arabic

Let-Mi: An Arabic Levantine Twitter Dataset for Misogynistic Language

Are They our Brothers? Analysis and Detection of Religious Hate Speech in the Arabic Twittersphere

Multilingual and Multi-Aspect Hate Speech Analysis (Arabic)

L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language

Abusive Language Detection on Arabic Social Media (Twitter)

Abusive Language Detection on Arabic Social Media (Al Jazeera)

Dataset Construction for the Detection of Anti-Social Behaviour in Online Communication in Arabic

Bengali

Hate Speech Detection in the Bengali language: A Dataset and its Baseline Evaluation

Chinese

SWSR: A Chinese Dataset and Lexicon for Online Sexism Detection

Croatian

CoRAL: a Context-aware Croatian Abusive Language Dataset

Datasets of Slovene and Croatian Moderated News Comments

Automating News Comment Moderation with Limited Resources: Benchmarking in Croatian and Estonian

Danish

Offensive Language and Hate Speech Detection for Danish

BAJER: Misogyny in Danish

Dutch

The Dutch Abusive Language Corpus v1.0 (DALC v1.0)

English

Not All Counterhate Tweets Elicit the Same Replies: A Fine-Grained Analysis

Pinpointing Fine-Grained Relationships between Hateful Tweets and Replies

Large-Scale Hate Speech Detection with Cross-Domain Transfer

ConvAbuse

Measuring Hate Speech

Learning From the Worst (Dynamically generated hate speech dataset)

The ‘Call me sexist, but’ sexism dataset

Hate Towards the Political Opponent: A Twitter Corpus Study of the 2020 US Elections on the Basis of Offensive Speech and Stance Detection__

AbuseEval v1.0

Do You Really Want to Hurt Me? Predicting Abusive Swearing in Social Media

Multimodal Meme Dataset (MultiOFF) for Identifying Offensive Content in Image and Text

Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hate

HateCheck: Functional Tests for Hate Speech Detection Models

Semeval-2021 Task 5: Toxic Spans Detection

ToxiSpanSE: An Explainable Toxicity Detection in Code Review Comments

Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech

HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection

ALONE: A Dataset for Toxic Behavior among Adolescents on Twitter

Towards a Comprehensive Taxonomy and Large-Scale Annotated Corpus for Online Slur Usage

Multimodal Meme Dataset (MultiOFF) for Identifying Offensive Content in Image and Text

Predicting the Type and Target of Offensive Posts in Social Media

Nuanced metrics for measuring unintended bias with real data for text classification

Introducing CAD: the Contextual Abuse Dataset

Automated Hate Speech Detection and the Problem of Offensive Language

Hate Speech Dataset from a White Supremacy Forum

Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter

Detecting Online Hate Speech Using Context Aware Models

The Gab Hate Corpus: A collection of 27k posts annotated for hate speech

Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter

When Does a Compliment Become Sexist? Analysis and Classification of Ambivalent Sexism Using Twitter Data

Overview of the Task on Automatic Misogyny Identification at IberEval 2018 (English)

CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech (English)

Characterizing and Detecting Hateful Users on Twitter

A Benchmark Dataset for Learning to Intervene in Online Hate Speech (Gab)

A Benchmark Dataset for Learning to Intervene in Online Hate Speech (Reddit)

Multilingual and Multi-Aspect Hate Speech Analysis (English)

Exploring Hate Speech Detection in Multimodal Publications

Predicting the Type and Target of Offensive Posts in Social Media

hatEval, SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter (English)

Peer to Peer Hate: Hate Speech Instigators and Their Targets

Overview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European Languages

Detecting East Asian Prejudice on Social media

Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior

A Large Labeled Corpus for Online Harassment Research

Ex Machina: Personal Attacks Seen at Scale, Personal attacks

Ex Machina: Personal Attacks Seen at Scale, Toxicity

Detecting cyberbullying in online communities (World of Warcraft)

Detecting cyberbullying in online communities (League of Legends)

A Quality Type-aware Annotated Corpus and Lexicon for Harassment Research

Ex Machina: Personal Attacks Seen at Scale, Aggression and Friendliness

Are Chess Discussions Racist? An Adversarial Hate Speech Data Set

ETHOS: an Online Hate Speech Detection Dataset (Binary)

ETHOS: an Online Hate Speech Detection Dataset (Multi label)

Twitter Sentiment Analysis

Toxicity Detection in Software Engineering: Automated Identification of Toxic Code Reviews Using ToxiCR

Toxicity Detection: Does Context Really Matter? CAT-LARGE (No Context)

Toxicity Detection: Does Context Really Matter? CAT-LARGE (With Context)

Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for Identifying and Classifying Hate in Online News Media

Estonian

Automating News Comment Moderation with Limited Resources: Benchmarking in Croatian and Estonian

HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection

French

CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech (French)

Multilingual and Multi-Aspect Hate Speech Analysis (French)

CyberAgressionAdo-v1

German

DeTox: A Comprehensive Dataset for German Offensive Language and Conversation Analysis

RP-Mod & RP-Crowd: Moderator- and Crowd-Annotated German News Comment Datasets

Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis

Detecting Offensive Statements Towards Foreigners in Social Media

GermEval 2018

Overview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European Languages

GAHD: A German Adversarial Hate Speech Dataset

Greek

Deep Learning for User Comment Moderation, Flagged Comments

Deep Learning for User Comment Moderation, Moderated Comments

Offensive Language Identification in Greek

Hindi / Hindi-English

Hostility Detection Dataset in Hindi

Aggression-annotated Corpus of Hindi-English Code-mixed Data

Aggression-annotated Corpus of Hindi-English Code-mixed Data

Did You Offend Me? Classification of Offensive Tweets in Hinglish Language

A Dataset of Hindi-English Code-Mixed Social Media Text for Hate Speech Detection

Overview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identification in Indo-European Languages

Indonesian

Hate Speech Detection in the Indonesian Language: A Dataset and Preliminary Study

Multi-Label Hate Speech and Abusive Language Detection in Indonesian Twitter

A Dataset and Preliminaries Study for Abusive Language Detection in Indonesian Social Media

Korean

BEEP! Korean Corpus of Online News Comments for Toxic Speech Detection__

Latvian

Latvian newspaper user comment dataset

Italian

An Italian Twitter Corpus of Hate Speech against Immigrants

Overview of the EVALITA 2018 Hate Speech Detection Task (Facebook)

Overview of the EVALITA 2018 Hate Speech Detection Task (Twitter)

Automatic Misogyny Identification (AMI) at Evalita 2020

CONAN - COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech (Italian)

Creating a WhatsApp Dataset to Study Pre-teen Cyberbullying

Polish

Results of the PolEval 2019 Shared Task 6:First Dataset and Open Shared Task for Automatic Cyberbullying Detection in Polish Twitter

Portuguese

Toxic Language Dataset for Brazilian Portuguese (ToLD-Br)

A Hierarchically-Labeled Portuguese Hate Speech Dataset

Offensive Comments in the Brazilian Web: A Dataset and Baseline Results

Offensive Language Identification Dataset for Brazilian Portuguese (OLID-BR)

Russian

Automatic Toxic Comment Detection in Social Media for Russian

Reducing Unintended Identity Bias in Russian Hate Speech Detection

Detection of Abusive Speech for Mixed Sociolects of Russian and Ukrainian Languages

Russian South Park

Slovene

Datasets of Slovene and Croatian Moderated News Comments

Spanish

Overview of MEX-A3T at IberEval 2018: Authorship and Aggressiveness Analysis in Mexican Spanish Tweets

Overview of the Task on Automatic Misogyny Identification at IberEval 2018 (Spanish)

hatEval, SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter (Spanish)

Turkish

Large-Scale Hate Speech Detection with Cross-Domain Transfer

A Corpus of Turkish Offensive Language on Social Media

Ukranian

Detection of Abusive Speech for Mixed Sociolects of Russian and Ukrainian Languages

Urdu

Hate-Speech and Offensive Language Detection in Roman Urdu


Lists of abusive keywords

  1. The Weaponized Word
    • “The Weaponized Word offers several thousand discriminatory, derogatory and threatening terms across 125+ languages, available through a RESTful API. Access is free for most academic researchers and registered humanitarian nonprofits.”
    • Data link: weaponizedword.org
  2. Hurtlex
  3. Gorrell et al.
  4. Wiegand et al.
  5. Chandrasekharan et al.
  6. Jiang et al.

How to Contribute

We accept entries to our catalogue based on pull requests to the README.md file. The dataset must be avaliable for download to be included in the list. If you want to add an entry, follow these steps!

Pasted Graphic

#### Title
* Link to publication: [url](url) - link to the documentation and/or a data statement about the data
* Link to data: [url](url) - direct download is preferred, e.g. a link straight to a .zip file
* Task description: How the task is framed in this data, e.g. "Binary (Hate, Not)", "Hierarchical", "Three-class (Hate speech, Offensive language, None)"
* Details of task: Free-text description of the task this data models, e.g. "Misogyny detection on social media in Danish"
* Size of dataset: Give the number of instances of abusive/non-abusive/other items
* Percentage abusive: e.g. 1.2%
* Language: e.g. Arabic
* Level of annotation: What is an "instance", in this dataset? e.g. Posts, User, Conversation, ... 
* Platform: e.g. twitter, snapchat, ..
* Medium: text / image / audio / ...
* Reference: Give a bibliographic reference for the data (if there is one), with title, author, year, venue etc

Pasted Graphic 1


This page is http://hatespeechdata.com/.