ElecBench: A Power Grid Dispatch Evaluation Benchmark for Large Language Models

In response to the challenges of grid stability, renewable energy integration, and electricity market dynamics, power grid dispatch is increasingly adopting large language models (LLMs) for their potential to improve efficiency and intelligence. However, the lack of specific performance benchmarks has limited their effective application. To fill this gap, we introduce "ElecBench", an evaluation benchmark designed for LLMs in power grid dispatch. ElecBench comprehensively covers sector-specific scenarios and deepens the testing of professional knowledge, focusing on six primary metrics: factuality, logicality, stability, security, fairness, and expressiveness, divided into 24 sub-metrics. This framework offers detailed insights into the capabilities and limitations of LLMs in power grid dispatch. We publicly release the test set and evaluate eight LLMs across various scenarios and metrics. ElecBench aims to become the standard benchmark for LLM applications in power grid dispatch, encouraging continuous updates and driving technological progress.