TIANWEN: A Comprehensive Benchmark for Evaluating LLMs in Chinese Classical Poetry Understanding and Reasoning

Chinese classical poetry, with its rich cultural heritage and intricate linguistic features, presents significant challenges for natural language processing systems. Despite recent advancements in Large Language Models (LLMs), their abilities in the domain of Chinese classical poetry remain largely underexplored. To bridge this gap, we introduce TianWen, a comprehensive benchmark designed to evaluate LLMs’ understanding and reasoning capabilities with respect to Chinese classical poetry. TianWen consists of 4k test instances across four understanding tasks and four reasoning tasks, incorporating varying levels of granularity and diverse task formats to enable thorough and multifaceted evaluation. We extensively evaluated 16 representative LLMs, including open-source and closed-source models on our benchmark, and the results indicate substantial room for improvement. Further analysis confirms the validity of scaling laws and highlights the challenges of Chinese classical poetry processing. We release our dataset and evaluation script at https://github.com/pzwstudy/TianWen-benchmark.