A Practical Black-box Adversarial Attack on Graph-learning-based Models for Malware Detection

Analyzing malware based on API call sequences is effective, as these sequences reflect the dynamic execution behavior of malware. The latest advancements in deep learning have facilitated the use of these techniques to mine useful information from API call sequences. However, existing methods primarily focus on extracting features from raw sequences, which may not capture critical information effectively, particularly in multi-process malware due to the interleaving issue of API calls. Recent research has introduced a graph-based method for embedding malware detection APIs. This approach offers two main advantages over existing embedding methods: graph models are invariant and can capture intra-process and inter-process behaviors more concisely and accurately. Given the inherent susceptibility of Graph Neural Networks (GNNs) to adversarial attacks, these models are vulnerable to such threats. To evaluate the security risks of existing malicious software detection systems based on API call graphs under black box settings, a heuristic intent masking algorithm has been designed to alter intentions that capture fine-grained malware behavior. This involves an iterative selection of important nodes based on an importance measure between nodes, followed by masking these nodes using heuristic paths. The malware file is then reconstructed based on an optimized transformation sequence that maintains the original file's semantics, adhering to program format constraints. Our approach systematically evaluates four state-of-the-art graph-based malware detection systems under a black-box setup. The evaluation results indicate that it achieves a high attack success rate. Additionally, the adversarial samples generated are able to maintain their original functionality.