Lexicalized second-order-HMM for ambiguity resolution in Chinese segmentation and POS tagging

Hidden Markov Model (HMM) is a main solution to ambiguities in Chinese segmentation and POS (part-of-speech) tagging. While most previous works for HMM-based Chinese segmentation and POS tagging consult POS information in contexts, they do not utilize lexical information which is crucial for resolving certain morphological ambiguity. This paper proposes a method which incorporates lexical information and wider context information into HMM. Model induction and related smoothing technique are presented in detail. Experiments indicate that this technique improves the segmentation and tagging accuracy by nearly 1%.