Semantic-Integrated Online Audit Log Reduction for Efficient Forensic Analysis

Audit logs are crucial for revealing and tracking sophisticated cyber threats due to their abundant system-level information. However, the immense scale of logs burdens storage resources and limits their lifecycle to days, which is insufficient for tracking multi-step attacks over months or years. Although log reduction techniques that cater to cold storage can mitigate this issue, many of these are restricted to offline batch processing in data centers. This incurs significant storage and transmission costs at endpoints. Moreover, many log reduction techniques fail to yield a suitable pattern for forensic analysis, which aims to identify signs of malicious activities by scrutinizing past events. In this paper, we present Sopr, an online audit log reduction technique designed to preserve traceability. Sopr enables real-time execution of the entire process, allowing reduction to be performed on raw log data streams. Specifically, our approach can effectively reduce events that lack causal dependence and involve repeated dependency relationships. To achieve this objective, we design a dual-cache architecture that simultaneously models semantically similar files and utilizes a versioned graph to preserve causality between log events. The synergy of these two components enhances the effectiveness of Sopr in log reduction. Our experiments on the DARPA TC datasets show that Sopr can achieve comparable event reduction factor in an online fashion to state-of-the-art offline approaches. Moreover, the runtime overhead and forensic analysis validity meet the deployment requirements for real-world environments.