Spectrum-BERT: Pretraining of Deep Bidirectional Transformers for Spectral Classification of Chinese Liquors

Counterfeit Chinese liquor incidents in China have significantly disrupted market order and jeopardized the health of consumers. Currently, deep learning-based spectral detection techniques are extensively leveraged in noninvasive food inspection. Excessive reliance on labels severely limits its application in real scenarios. To make better use of limited samples, we are the first to use the 'unsupervised pretraining & supervised fine-tuning' paradigm in combining the Transformer architecture for feature extraction and classification of the Chinese liquor spectrum, and propose Spectrum-BERT, which represents Bidirectional Encoder Representations from Transformers for Spectrum. Specifically, we creatively propose spectral curve partitioning and 1-D convolutional layer mapping to maintain the model's sensitivity to characteristic peak locations and local information of spectral curves. Moreover, the paradigm of 'unsupervised pretraining & supervised fine-tuning' addresses the limitation of label deficiency, thus improving the model's applicability. Finally, we have conducted extensive experiments on the real liquor spectral dataset. Comparative experiments demonstrate that Spectrum-BERT outperforms all baselines on all metrics using only 70% of supervised signal. The limit experimental results show that Spectrum-BERT can still maintain its lead using only the 10% supervised signal. Thanks to the more efficient model architecture, Spectrum-BERT's model parameters and FLOPs are only 1/568 and 1/322 of those of baselines.