Will Sentiment Extraction based on ChatGPT yield Better Predictive Outcomes? Evidence from an Online Travel Agency

The rise of Large Language Models (LLMs) has significantly advanced natural language processing, outperforming many traditional tools such as rule-based systems and machine learning models. ChatGPT, a leading example, has exhibited exceptional capabilities in textual analysis. This study examines whether ChatGPT can outperform traditional sentiment analysis methods in the context of sales prediction leveraging online review data from online travel agencies, Booking and Expedia. We employ review ratings, and sentiment analysis tools, including VADER, RoBERTa, and ChatGPT, to predict revenue metrics. We find that both VADER and RoBERTa exhibit comparable predictive power to review ratings, whereas ChatGPT's sentiment scores demonstrate a weaker correlation with revenue metrics. Grounded in Heuristic-Systematic Models (HSM) from dual process theory, we posit that customers rely predominantly on heuristic cues (review ratings and keywords of extreme words) for decision making, which are better captured by traditional sentiment analysis tools. In contrast, ChatGPT’s evaluation, which emphasizes systematic review content processing, aligns less with consumer behavior in this context. This study contributes theoretically to extending HSM to illustrate how AIGC moderates systematic information processing in sales prediction. It also offers empirical insights into the comparative effectiveness of sentiment analysis tools, providing a practical implication for e-commerce platforms and managers regarding the adoption of AIGC in strategic decision-making. Caution is advised when integrating AIGC into sales and operational strategies.