LPFHE: Low-Complexity Polynomial CNNs for Secure Inference over FHE

Machine learning as a service (MLaaS) has gained popularity among clients. To address the privacy concerns in MLaaS, fully homomorphic encryption (FHE) has been introduced to protect clients’ data. However, FHE cannot directly evaluate the non-arithmetic activation function in convolutional neural networks (CNNs). Existing works replace the activation function with polynomials of varying degrees to obtain FHE-friendly CNNs, while having to face a trade-off between accuracy loss and latency increases. It remains a significant challenge to maintain accuracy with low latency in secure inference. We propose a framework called LPFHE to precisely approximate the essential ReLU function in CNNs using low-complexity polynomials. LPFHE supports finding the optimal approximation domain and polynomial for each ReLU function. By integrating our segmented weighted least squares algorithm with the Remez algorithm, LPFHE achieves higher approximation precision compared to existing works. Consequently, LPFHE is capable of generating a low-complexity polynomial CNN with high inference accuracy, as the low-degree polynomials preserve the properties of the ReLU function well.    We implement LPFHE on ResNet20/32/44 networks on encrypted CIFAR10/100 datasets under RNS-CKKS, which shows up to a 48.7% reduction in amortized inference latency with little accuracy loss, in comparison to the previous works.