ZAIF: A Zero-Shot Anomaly Inference Framework for Fire Detection and Segmentation with Multi-modal Data

Fire detection, as a key technology to ensure social security, has a direct impact on fire response efficiency and loss control in terms of its accuracy and real-time performance. Although existing deep learning methods such as YOLO and Faster R-CNN have achieved remarkable results in the field of fire detection, they generally suffer from the problems of high data dependency and limited generalization ability. To address these limitations, this paper proposes an innovative multimodal zero-sample fire detection method (ZAIF). The method extracts fire features by fusing the complementary information of visible and infrared images, combining with the background segmentation technique, and realizes multi-scale feature embedding by using a GEM image encoder. Meanwhile, ZAIF innovatively integrates the CLIP text encoder and the refined text cues generated by the Large Language Model (LLM) to realize the deep fusion of semantic and visual features. Experimental results show that the classification and positioning accuracy of ZAIF on the self-collected combustion experimental dataset is significantly better than the existing zero-shot detection method. In addition, when the number of samples is small, ZAIF shows better detection performance compared with YOLOv11, a very advanced method in the field of image detection, providing a new solution for fire detection in practical application scenarios.