Research on multiple layer text topics identification algorithm based on the dynamic diverse thresholds clustering

In many NLP applications, text topic identification is a common problem. Traditional topic identification method always generated a single-layered topic structure which is usually inaccurate topic division even if generated manually by the human experts. This paper proposed a concept of multi-layer topic. Secondly, this paper proposed an iterative text units clustering method to recognize automatically the hierarchical topic of the text set. In this method, text clustering processing paused when each topic in the text set were correctly divided into multiple sub-topics, and such processing continued until a hierarchical topic tree had been built. A key problem that automatically selected multiple pause threshold values was resolved by the minimized clustering entropy method in this paper. The results of experiments demonstrated the effectiveness of the method.