CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension

Xingyi Duan; Baoxin Wang; Ziyue Wang; Wentao Ma; Yiming Cui; Dayong Wu; Shijin Wang; Ting Liu; Tianxiang Huo; Zhen Hu; Heng Wang; Zhiyuan Liu

doi:10.1007/978-3-030-32381-3_36

CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension

Xingyi Duan^*
, Baoxin Wang
, Ziyue Wang
, Wentao Ma
, Yiming Cui
, Dayong Wu
, Shijin Wang
, Ting Liu
, Tianxiang Huo
, Zhen Hu
, Heng Wang
, Zhiyuan Liu

^*Corresponding author for this work

Faculty of Computing

IFLYTEK Co., Ltd.
Harbin Institute of Technology
China Justice Big Data Institute
Tsinghua University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

71 Scopus citations

Abstract

We present a Chinese judicial reading comprehension (CJRC) dataset which contains approximately 10K documents and almost 50K questions with answers. The documents come from judgment documents and the questions are annotated by law experts. The CJRC dataset can help researchers extract elements by reading comprehension technology. Element extraction is an important task in the legal field. However, it is difficult to predefine the element types completely due to the diversity of document types and causes of action. By contrast, machine reading comprehension technology can quickly extract elements by answering various questions from the long document. We build two strong baseline models based on BERT and BiDAF. The experimental results show that there is enough space for improvement compared to human annotators.

Original language	English
Title of host publication	Chinese Computational Linguistics - 18th China National Conference, CCL 2019, Proceedings
Editors	Maosong Sun, Yang Liu, Zhiyuan Liu, Xuanjing Huang, Heng Ji
Publisher	Springer
Pages	439-451
Number of pages	13
ISBN (Print)	9783030323806
DOIs	https://doi.org/10.1007/978-3-030-32381-3_36
State	Published - 2019
Event	18th China National Conference on Computational Linguistics, CCL 2019 - Kunming, China Duration: 18 Oct 2019 → 20 Oct 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11856 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	18th China National Conference on Computational Linguistics, CCL 2019
Country/Territory	China
City	Kunming
Period	18/10/19 → 20/10/19

Access to Document

10.1007/978-3-030-32381-3_36

Cite this

Duan, X., Wang, B., Wang, Z., Ma, W., Cui, Y., Wu, D., Wang, S., Liu, T., Huo, T., Hu, Z., Wang, H., & Liu, Z. (2019). CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension. In M. Sun, Y. Liu, Z. Liu, X. Huang, & H. Ji (Eds.), Chinese Computational Linguistics - 18th China National Conference, CCL 2019, Proceedings (pp. 439-451). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11856 LNAI). Springer. https://doi.org/10.1007/978-3-030-32381-3_36

Duan, Xingyi ; Wang, Baoxin ; Wang, Ziyue et al. / CJRC : A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension. Chinese Computational Linguistics - 18th China National Conference, CCL 2019, Proceedings. editor / Maosong Sun ; Yang Liu ; Zhiyuan Liu ; Xuanjing Huang ; Heng Ji. Springer, 2019. pp. 439-451 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{0057b7d16260491baa1b7e07a6f9f651,

title = "CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension",

abstract = "We present a Chinese judicial reading comprehension (CJRC) dataset which contains approximately 10K documents and almost 50K questions with answers. The documents come from judgment documents and the questions are annotated by law experts. The CJRC dataset can help researchers extract elements by reading comprehension technology. Element extraction is an important task in the legal field. However, it is difficult to predefine the element types completely due to the diversity of document types and causes of action. By contrast, machine reading comprehension technology can quickly extract elements by answering various questions from the long document. We build two strong baseline models based on BERT and BiDAF. The experimental results show that there is enough space for improvement compared to human annotators.",

author = "Xingyi Duan and Baoxin Wang and Ziyue Wang and Wentao Ma and Yiming Cui and Dayong Wu and Shijin Wang and Ting Liu and Tianxiang Huo and Zhen Hu and Heng Wang and Zhiyuan Liu",

note = "Publisher Copyright: {\textcopyright} 2019, Springer Nature Switzerland AG.; 18th China National Conference on Computational Linguistics, CCL 2019 ; Conference date: 18-10-2019 Through 20-10-2019",

year = "2019",

doi = "10.1007/978-3-030-32381-3\_36",

language = "英语",

isbn = "9783030323806",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "439--451",

editor = "Maosong Sun and Yang Liu and Zhiyuan Liu and Xuanjing Huang and Heng Ji",

booktitle = "Chinese Computational Linguistics - 18th China National Conference, CCL 2019, Proceedings",

address = "德国",

}

Duan, X, Wang, B, Wang, Z, Ma, W, Cui, Y, Wu, D, Wang, S, Liu, T, Huo, T, Hu, Z, Wang, H & Liu, Z 2019, CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension. in M Sun, Y Liu, Z Liu, X Huang & H Ji (eds), Chinese Computational Linguistics - 18th China National Conference, CCL 2019, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11856 LNAI, Springer, pp. 439-451, 18th China National Conference on Computational Linguistics, CCL 2019, Kunming, China, 18/10/19. https://doi.org/10.1007/978-3-030-32381-3_36

CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension. / Duan, Xingyi; Wang, Baoxin; Wang, Ziyue et al.
Chinese Computational Linguistics - 18th China National Conference, CCL 2019, Proceedings. ed. / Maosong Sun; Yang Liu; Zhiyuan Liu; Xuanjing Huang; Heng Ji. Springer, 2019. p. 439-451 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11856 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - CJRC

T2 - 18th China National Conference on Computational Linguistics, CCL 2019

AU - Duan, Xingyi

AU - Wang, Baoxin

AU - Wang, Ziyue

AU - Ma, Wentao

AU - Cui, Yiming

AU - Wu, Dayong

AU - Wang, Shijin

AU - Liu, Ting

AU - Huo, Tianxiang

AU - Hu, Zhen

AU - Wang, Heng

AU - Liu, Zhiyuan

PY - 2019

Y1 - 2019

N2 - We present a Chinese judicial reading comprehension (CJRC) dataset which contains approximately 10K documents and almost 50K questions with answers. The documents come from judgment documents and the questions are annotated by law experts. The CJRC dataset can help researchers extract elements by reading comprehension technology. Element extraction is an important task in the legal field. However, it is difficult to predefine the element types completely due to the diversity of document types and causes of action. By contrast, machine reading comprehension technology can quickly extract elements by answering various questions from the long document. We build two strong baseline models based on BERT and BiDAF. The experimental results show that there is enough space for improvement compared to human annotators.

AB - We present a Chinese judicial reading comprehension (CJRC) dataset which contains approximately 10K documents and almost 50K questions with answers. The documents come from judgment documents and the questions are annotated by law experts. The CJRC dataset can help researchers extract elements by reading comprehension technology. Element extraction is an important task in the legal field. However, it is difficult to predefine the element types completely due to the diversity of document types and causes of action. By contrast, machine reading comprehension technology can quickly extract elements by answering various questions from the long document. We build two strong baseline models based on BERT and BiDAF. The experimental results show that there is enough space for improvement compared to human annotators.

UR - https://www.scopus.com/pages/publications/85075712647

U2 - 10.1007/978-3-030-32381-3_36

DO - 10.1007/978-3-030-32381-3_36

M3 - 会议稿件

AN - SCOPUS:85075712647

SN - 9783030323806

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 439

EP - 451

BT - Chinese Computational Linguistics - 18th China National Conference, CCL 2019, Proceedings

A2 - Sun, Maosong

A2 - Liu, Yang

A2 - Liu, Zhiyuan

A2 - Huang, Xuanjing

A2 - Ji, Heng

PB - Springer

Y2 - 18 October 2019 through 20 October 2019

ER -

Duan X, Wang B, Wang Z, Ma W, Cui Y, Wu D et al. CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension. In Sun M, Liu Y, Liu Z, Huang X, Ji H, editors, Chinese Computational Linguistics - 18th China National Conference, CCL 2019, Proceedings. Springer. 2019. p. 439-451. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-32381-3_36

CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this