This page contains detailed guidelines for NLPCC 2021 Shared Task 7 - Fine-Grain Dialogue Social Bias Measurement.
This task aims to measure the social bias in dialogue scenario. Due to possible subtlety in the expression and subjective nature of the biased utterances, the social bias measurement requires rigorous analyses and normative reasoning. Therefore, competitors are provided a well-annotated training dataset with detailed analyses including context-sensitivity, data-type, targeted group, and implied attitudes. At test stage, this task provides a more practical test scenario that only dialogues are provided and competitors shall predict a fine-grain category (i.e. irrelevant, anti-bias, neutral, and biased) w.r.t. dialogue social bias.
Organizers: Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, Helen M. Meng
Important Dates
- April 6, 2022: Training data is avaliable at link.
- May 5, 2022: Registration deadline. The registration information is recorded at link, please check if your information is recoreded correctly and contact us if any problems occur.
- May 10, 2022: Test data is avaliable at link.
- May 20, 2022: Results submission deadline.
Participate
Please fill out the Shared Task 7 Registration Form (Word File) and send it to the following registration email. Registration Email: jyzhou@se.cuhk.edu.hk
Detailed Dataset Descriptions and Baselines
http://arxiv.org/abs/2202.08011 (We refine the annotations and construct CDial-Bias Dataset 2.0 for this shared task. The statistics and baseline performances may differ to some extend.)
Task:
| Goal | measure the social bias in dialogue scenario. |
|---|---|
| Input | A 2-turn dialogue |
| Output | A fine-grain social bias label of the second dialogue turn. (i.e. 0 - Irrelevant, 1 - Anti-bias, 2 - Neutral, and 3 - Biased). |
Dataset:
Format : The Cdial-Bias Dataset 2.0 has follwoing entries.
| Explaination | |
|---|---|
| Q | Dialogturn turn 1. |
| A | Dialogturn turn 2. |
| Topic | The topic of the dialogue, including Race, Gender, Region, Occupatioin. |
| Context Sensitivity | 0 - Context-independent; 1 - Context-sensitive. |
| Data Type | 0 - Irrelevant; 1 - Bias-expressing; 2 - Bias-discussing. |
| Bias Attitudes | 0 - NA (Irrelevant data); 1 - Anti-Bias; 2 - Neutral; 3 - Biased. |
| Referrenced Groups | Presented in freetext. Multiple groups are splited by '/'. |
Statistics: Detailed statistics can be found at link
Evaluation
Evaluation metric: Macro F1 score on the test set.
Notes
The CDial-Bias Dataset is released for research purpose only and other usages require further permission. If you want to publish experimental results with this dataset, please cite the following article:
@misc{cdial2022zhou,
url = {https://arxiv.org/abs/2202.08011},
author = {Zhou, Jingyan and Deng, Jiawen and Mi, Fei and Li, Yitong and Wang, Yasheng and Huang, Minlie and Jiang, Xin and Liu, Qun and Meng, Helen},
title = {Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarks},
publisher = {arXiv},
year = {2022}
}