Skip to main navigation Skip to search Skip to main content

A similarity-based approach to data sparseness problem of Chinese language modeling

  • Jinghui Xiao*
  • , Bingquan Liu
  • , Xiaolong Wang
  • , Bing Li
  • *Corresponding author for this work
  • School of Computer Science and Technology, Harbin Institute of Technology

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Data sparseness problem is inherent and severe in language modeling. Smoothing techniques are usually widely used to solve this problem. However, traditional smoothing techniques are all based on statistical hypotheses without concerning about linguistic knowledge. This paper introduces semantic information into smoothing technique and proposes a similarity-based smoothing method which is based on both statistical hypothesis and linguistic hypothesis. An experiential iterative algorithm is presented to optimize system parameters. Experiment results prove that compared with traditional smoothing techniques, our method can greatly improve the performance of language model.

Original languageEnglish
Title of host publicationMICAI 2005
Subtitle of host publicationAdvances in Artificial Intelligence - 4th Mexican International Conference on Artificial Intelligence, Proceedings
Pages761-769
Number of pages9
DOIs
StatePublished - 2005
Externally publishedYes
Event4th Mexican International Conference on Artificial Intelligence, MICAI 2005 - Monterrey, Mexico
Duration: 14 Nov 200518 Nov 2005

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume3789 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference4th Mexican International Conference on Artificial Intelligence, MICAI 2005
Country/TerritoryMexico
CityMonterrey
Period14/11/0518/11/05

Fingerprint

Dive into the research topics of 'A similarity-based approach to data sparseness problem of Chinese language modeling'. Together they form a unique fingerprint.

Cite this