How unsupervised learning affects character tagging based chinese word segmentation: A quantitative investigation

Yan Song; Chunyu Kit; Ruifeng Xu; Hai Zhao

doi:10.1109/ICMLC.2009.5212769

How unsupervised learning affects character tagging based chinese word segmentation: A quantitative investigation

Yan Song^*
, Chunyu Kit
, Ruifeng Xu
, Hai Zhao

^*Corresponding author for this work

City University of Hong Kong

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Integrating global information of unsupervised segmentation into Conditional Random Fields (CRF) learning has been proved effective to enhance the performance of the character tagging based Chinese Word Segmentation. By comparing CRF models with and without unsupervised learning enhancement, we investigate how unsupervised learning affects the performance. Especially, two kinds of segmented words, in-vocabulary and out-of-vocabulary words, are separately analyzed case by case to see what part of those words are affected by unsupervised learning. In addition, the cost of the additional features derived from unsupervised segmentation are also taken into account and evaluated.

Original language	English
Title of host publication	Proceedings of the 2009 International Conference on Machine Learning and Cybernetics
Publisher	IEEE Computer Society
Pages	3481-3486
Number of pages	6
ISBN (Print)	9781424437030
DOIs	https://doi.org/10.1109/ICMLC.2009.5212769
State	Published - 2009
Externally published	Yes
Event	8th International Conference on Machine Learning and Cybernetics, ICMLC 2009 - Baoding, China Duration: 12 Jul 2009 → 15 Jul 2009

Publication series

Name	Proceedings of the 2009 International Conference on Machine Learning and Cybernetics
Volume	6

Conference

Conference	8th International Conference on Machine Learning and Cybernetics, ICMLC 2009
Country/Territory	China
City	Baoding
Period	12/07/09 → 15/07/09

Keywords

Chinese word segmentation
Frequent substring extraction
In-vocabulary words
Out-of-vocabulary words
Unsupervised learning

Access to Document

10.1109/ICMLC.2009.5212769

Cite this

Song, Y., Kit, C., Xu, R., & Zhao, H. (2009). How unsupervised learning affects character tagging based chinese word segmentation: A quantitative investigation. In Proceedings of the 2009 International Conference on Machine Learning and Cybernetics (pp. 3481-3486). Article 5212769 (Proceedings of the 2009 International Conference on Machine Learning and Cybernetics; Vol. 6). IEEE Computer Society. https://doi.org/10.1109/ICMLC.2009.5212769

Song, Yan ; Kit, Chunyu ; Xu, Ruifeng et al. / How unsupervised learning affects character tagging based chinese word segmentation : A quantitative investigation. Proceedings of the 2009 International Conference on Machine Learning and Cybernetics. IEEE Computer Society, 2009. pp. 3481-3486 (Proceedings of the 2009 International Conference on Machine Learning and Cybernetics).

@inproceedings{8295c1f02d8a4d1c85ddf626242a7cc2,

title = "How unsupervised learning affects character tagging based chinese word segmentation: A quantitative investigation",

abstract = "Integrating global information of unsupervised segmentation into Conditional Random Fields (CRF) learning has been proved effective to enhance the performance of the character tagging based Chinese Word Segmentation. By comparing CRF models with and without unsupervised learning enhancement, we investigate how unsupervised learning affects the performance. Especially, two kinds of segmented words, in-vocabulary and out-of-vocabulary words, are separately analyzed case by case to see what part of those words are affected by unsupervised learning. In addition, the cost of the additional features derived from unsupervised segmentation are also taken into account and evaluated.",

keywords = "Chinese word segmentation, Frequent substring extraction, In-vocabulary words, Out-of-vocabulary words, Unsupervised learning",

author = "Yan Song and Chunyu Kit and Ruifeng Xu and Hai Zhao",

year = "2009",

doi = "10.1109/ICMLC.2009.5212769",

language = "英语",

isbn = "9781424437030",

series = "Proceedings of the 2009 International Conference on Machine Learning and Cybernetics",

publisher = "IEEE Computer Society",

pages = "3481--3486",

booktitle = "Proceedings of the 2009 International Conference on Machine Learning and Cybernetics",

address = "美国",

note = "8th International Conference on Machine Learning and Cybernetics, ICMLC 2009 ; Conference date: 12-07-2009 Through 15-07-2009",

}

Song, Y, Kit, C, Xu, R & Zhao, H 2009, How unsupervised learning affects character tagging based chinese word segmentation: A quantitative investigation. in Proceedings of the 2009 International Conference on Machine Learning and Cybernetics., 5212769, Proceedings of the 2009 International Conference on Machine Learning and Cybernetics, vol. 6, IEEE Computer Society, pp. 3481-3486, 8th International Conference on Machine Learning and Cybernetics, ICMLC 2009, Baoding, China, 12/07/09. https://doi.org/10.1109/ICMLC.2009.5212769

How unsupervised learning affects character tagging based chinese word segmentation: A quantitative investigation. / Song, Yan; Kit, Chunyu; Xu, Ruifeng et al.
Proceedings of the 2009 International Conference on Machine Learning and Cybernetics. IEEE Computer Society, 2009. p. 3481-3486 5212769 (Proceedings of the 2009 International Conference on Machine Learning and Cybernetics; Vol. 6).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - How unsupervised learning affects character tagging based chinese word segmentation

T2 - 8th International Conference on Machine Learning and Cybernetics, ICMLC 2009

AU - Song, Yan

AU - Kit, Chunyu

AU - Xu, Ruifeng

AU - Zhao, Hai

PY - 2009

Y1 - 2009

N2 - Integrating global information of unsupervised segmentation into Conditional Random Fields (CRF) learning has been proved effective to enhance the performance of the character tagging based Chinese Word Segmentation. By comparing CRF models with and without unsupervised learning enhancement, we investigate how unsupervised learning affects the performance. Especially, two kinds of segmented words, in-vocabulary and out-of-vocabulary words, are separately analyzed case by case to see what part of those words are affected by unsupervised learning. In addition, the cost of the additional features derived from unsupervised segmentation are also taken into account and evaluated.

AB - Integrating global information of unsupervised segmentation into Conditional Random Fields (CRF) learning has been proved effective to enhance the performance of the character tagging based Chinese Word Segmentation. By comparing CRF models with and without unsupervised learning enhancement, we investigate how unsupervised learning affects the performance. Especially, two kinds of segmented words, in-vocabulary and out-of-vocabulary words, are separately analyzed case by case to see what part of those words are affected by unsupervised learning. In addition, the cost of the additional features derived from unsupervised segmentation are also taken into account and evaluated.

KW - Chinese word segmentation

KW - Frequent substring extraction

KW - In-vocabulary words

KW - Out-of-vocabulary words

KW - Unsupervised learning

UR - https://www.scopus.com/pages/publications/70350741381

U2 - 10.1109/ICMLC.2009.5212769

DO - 10.1109/ICMLC.2009.5212769

M3 - 会议稿件

AN - SCOPUS:70350741381

SN - 9781424437030

T3 - Proceedings of the 2009 International Conference on Machine Learning and Cybernetics

SP - 3481

EP - 3486

BT - Proceedings of the 2009 International Conference on Machine Learning and Cybernetics

PB - IEEE Computer Society

Y2 - 12 July 2009 through 15 July 2009

ER -

Song Y, Kit C, Xu R, Zhao H. How unsupervised learning affects character tagging based chinese word segmentation: A quantitative investigation. In Proceedings of the 2009 International Conference on Machine Learning and Cybernetics. IEEE Computer Society. 2009. p. 3481-3486. 5212769. (Proceedings of the 2009 International Conference on Machine Learning and Cybernetics). doi: 10.1109/ICMLC.2009.5212769

How unsupervised learning affects character tagging based chinese word segmentation: A quantitative investigation

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this