IRePf: An Instruction Reorganization Virtual Platform for Kernel Stack Overflow Detection

Stack overflow vulnerabilities are among the most common security issues. However, the existing stack overflow detection solutions only protect the return address and ignore the imbalance between function calls and returns in the system, which will lead to a higher false-positive rate. In this paper, we propose an instruction reorganization virtual platform technique for kernel stack overflow detection, named IRePf. It can dynamically monitor the kernel stack when the system is running through dynamic reorganization instructions, thread creation and termination, call instructions, and RET instructions. IRePf uses backup stack creation and destruction technology to back up the return address and the address of the return address at the same time. IRePf determines whether the stack overflow occurs in the system when the function call and return are unbalanced to realize the kernel stack overflow detection. The experimental results show that IRePf can effectively detect stack overflow attacks, has low system resource occupancy and high real-time performance, and effectively improves the ability and security of defense stack attacks.