From metrics to ranking: A data-driven evaluation framework for threat intelligence platforms with real-world attack observations

Threat Intelligence Platforms (TIPs) play an essential role in proactive cybersecurity operations by systematically gathering, analyzing, and disseminating threat data, thereby offering critical insights into the evolving threat landscape. However, the significant variation in TIP quality and the inherent limitations of any single platform underscore the pressing need for a standardized methodology to effectively assess and compare TIP capabilities. In this paper, we propose a novel comprehensive evaluation framework that assesses TIPs from dual perspectives of platform performance and intelligence quality. The framework integrates eight core metrics to systematically measure detection capability, alert uniqueness, consistency, and information depth, while allowing customizable weight adjustment based on specific operational requirements. Furthermore, we introduce TIPRank, a manipulation-resistant ranking mechanism inspired by PageRank. By synthesizing composite quality scores with citation relationships via a weighted directed graph, TIPRank generates stable and well-balanced rankings. We evaluate seven commercial and open-source TIPs using two real-world malicious-IP datasets (768 and 17,120 IPs) derived from attack and honeypot logs to validate their detection capabilities and quantify performance disparities. Robustness analysis, including metric ablation, data sampling, and weight sensitivity experiments, confirms that the proposed framework produces consistent results across varied configurations. The experimental results provide actionable insights for scientifically grounded TIP selection and contribute to strengthening proactive cyber defense.