A self-adaptive lexicon construction algorithm for Chinese language modeling

The lexicon quality affects the performance of Chinese language model directly. However, the lexicon compilation is separated from Chinese language modeling, resulting in two severe problems: Firstly, the current language models can not achieve the optimal performance due to the limitation of lexicon scale; secondly, it is hard to apply the current language models to special areas due to the absence of lexicon. This paper aims to improve the performance of Chinese language model by constructing optimal lexicon. Meanwhile, it can self-adapt the area of training corpus automatically. Firstly, this paper combines the lexicon compilation with Chinese language modeling and proposes an iterative algorithm framework. Under this framework, it proposes the concept of character lexical significance (CLS) to describe Chinese lexical principle. Together with the statistical features, a multi-feature based algorithm is proposed for Chinese lexicon construction. Finally, it proposes two heuristic rules to adjust the parameters so as to self-adapt the area of training corpus. From the experimental results, it is found that the system can obtain the optimal Chinese lexicon as well as the high-performance Chinese language model. Moreover, the proposed techniques can self-adapt the area of training corpus successfully.