TextBrewer: An open-source knowledge distillation toolkit for natural language processing

Ziqing Yang; Yiming Cui; Zhipeng Chen; Wanxiang Che; Ting Liu; Shijin Wang; Guoping Hu

TextBrewer: An open-source knowledge distillation toolkit for natural language processing

Ziqing Yang
, Yiming Cui
, Zhipeng Chen
, Wanxiang Che
, Ting Liu
, Shijin Wang
, Guoping Hu

Faculty of Computing

IFLYTEK Co., Ltd.
Harbin Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

21 Scopus citations

Abstract

In this paper, we introduce TextBrewer, an open-source knowledge distillation toolkit designed for natural language processing. It works with different neural network models and supports various kinds of supervised learning tasks, such as text classification, reading comprehension, sequence labeling. TextBrewer provides a simple and uniform workflow that enables quick setting up of distillation experiments with highly flexible configurations. It offers a set of predefined distillation methods and can be extended with custom code. As a case study, we use TextBrewer to distill BERT on several typical NLP tasks. With simple configurations, we achieve results that are comparable with or even higher than the public distilled BERT models with similar numbers of parameters.

Original language	English
Title of host publication	ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the System Demonstrations
Publisher	Association for Computational Linguistics (ACL)
Pages	9-16
Number of pages	8
ISBN (Electronic)	9781952148040
State	Published - 2020
Event	58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 - Virtual, Online, United States Duration: 5 Jul 2020 → 10 Jul 2020

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	58th Annual Meeting of the Association for Computational Linguistics, ACL 2020
Country/Territory	United States
City	Virtual, Online
Period	5/07/20 → 10/07/20

Cite this

Yang, Z., Cui, Y., Chen, Z., Che, W., Liu, T., Wang, S., & Hu, G. (2020). TextBrewer: An open-source knowledge distillation toolkit for natural language processing. In ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the System Demonstrations (pp. 9-16). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL).

Yang, Ziqing ; Cui, Yiming ; Chen, Zhipeng et al. / TextBrewer : An open-source knowledge distillation toolkit for natural language processing. ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the System Demonstrations. Association for Computational Linguistics (ACL), 2020. pp. 9-16 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

@inproceedings{85d6b933551c4d828d5b29ff9ad8497b,

title = "TextBrewer: An open-source knowledge distillation toolkit for natural language processing",

abstract = "In this paper, we introduce TextBrewer, an open-source knowledge distillation toolkit designed for natural language processing. It works with different neural network models and supports various kinds of supervised learning tasks, such as text classification, reading comprehension, sequence labeling. TextBrewer provides a simple and uniform workflow that enables quick setting up of distillation experiments with highly flexible configurations. It offers a set of predefined distillation methods and can be extended with custom code. As a case study, we use TextBrewer to distill BERT on several typical NLP tasks. With simple configurations, we achieve results that are comparable with or even higher than the public distilled BERT models with similar numbers of parameters.",

author = "Ziqing Yang and Yiming Cui and Zhipeng Chen and Wanxiang Che and Ting Liu and Shijin Wang and Guoping Hu",

note = "Publisher Copyright: {\textcopyright} 2020 Association for Computational Linguistics; 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 ; Conference date: 05-07-2020 Through 10-07-2020",

year = "2020",

language = "英语",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "9--16",

booktitle = "ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the System Demonstrations",

address = "澳大利亚",

}

Yang, Z, Cui, Y, Chen, Z, Che, W , Liu, T, Wang, S & Hu, G 2020, TextBrewer: An open-source knowledge distillation toolkit for natural language processing. in ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the System Demonstrations. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 9-16, 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Virtual, Online, United States, 5/07/20.

TextBrewer: An open-source knowledge distillation toolkit for natural language processing. / Yang, Ziqing; Cui, Yiming; Chen, Zhipeng et al.
ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the System Demonstrations. Association for Computational Linguistics (ACL), 2020. p. 9-16 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - TextBrewer

T2 - 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020

AU - Yang, Ziqing

AU - Cui, Yiming

AU - Chen, Zhipeng

AU - Che, Wanxiang

AU - Liu, Ting

AU - Wang, Shijin

AU - Hu, Guoping

PY - 2020

Y1 - 2020

N2 - In this paper, we introduce TextBrewer, an open-source knowledge distillation toolkit designed for natural language processing. It works with different neural network models and supports various kinds of supervised learning tasks, such as text classification, reading comprehension, sequence labeling. TextBrewer provides a simple and uniform workflow that enables quick setting up of distillation experiments with highly flexible configurations. It offers a set of predefined distillation methods and can be extended with custom code. As a case study, we use TextBrewer to distill BERT on several typical NLP tasks. With simple configurations, we achieve results that are comparable with or even higher than the public distilled BERT models with similar numbers of parameters.

AB - In this paper, we introduce TextBrewer, an open-source knowledge distillation toolkit designed for natural language processing. It works with different neural network models and supports various kinds of supervised learning tasks, such as text classification, reading comprehension, sequence labeling. TextBrewer provides a simple and uniform workflow that enables quick setting up of distillation experiments with highly flexible configurations. It offers a set of predefined distillation methods and can be extended with custom code. As a case study, we use TextBrewer to distill BERT on several typical NLP tasks. With simple configurations, we achieve results that are comparable with or even higher than the public distilled BERT models with similar numbers of parameters.

UR - https://www.scopus.com/pages/publications/85100668163

M3 - 会议稿件

AN - SCOPUS:85100668163

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 9

EP - 16

BT - ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the System Demonstrations

PB - Association for Computational Linguistics (ACL)

Y2 - 5 July 2020 through 10 July 2020

ER -

Yang Z, Cui Y, Chen Z, Che W , Liu T, Wang S et al. TextBrewer: An open-source knowledge distillation toolkit for natural language processing. In ACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the System Demonstrations. Association for Computational Linguistics (ACL). 2020. p. 9-16. (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

TextBrewer: An open-source knowledge distillation toolkit for natural language processing

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this