Theoretical study of ReLU nonlinear activation functions in diffractive neural networks

This study investigates the impact of the classical rectified linear unit (ReLU) activation function and its derivatives on the inference capability of optical neural networks (ONNs) using a diffractive neural network platform, and it elucidates the relationship between the integration method of nonlinear functions into diffractive neural networks and their learning performance. The results demonstrate that while diffractive networks exhibit strong training adaptability, inappropriate nonlinear functions can compromise their performance. For instance, in a five-layer network, the addition of rectified translational linear unit (RTReLU) after each layer caused a drop in classification accuracy on the MNIST test set to 91.4% due to optical intensity attenuation, compared to the 92.6% attained by a purely linear network. In contrast, incorporating parametric rectified linear unit (PReLU) after each layer preserved post-threshold information, thereby improving the accuracy to 95.8%. In a three-layer network, however, the reduced depth mitigated optical intensity loss, and the resulting network with RTReLU added after each layer outperformed the linear network. Furthermore, it was found that strategic placement of a single RTReLU activation function can significantly enhance performance. For example, a five-layer network with RTReLU added only at the final layer achieved a peak accuracy of 96.6%.