Identifying and Removing the Ghosts of Reproducibility in Service Recommendation Research

A service recommendation system is an information system that helps build mashups quickly to implement new features in response to environmental changes. With the development of deep learning (DL) in recent years, more researchers have started using DL-based methods to solve service recommendation problem and have achieved remarkable results. However, these works have some common deficiencies on non-unified dataset, pre-trained model, evaluation protocol, and experiment environment. These issues will disrupt evaluating the performance of models accurately and make reproducing them difficult. To solve these problems, we propose a service mashup recommendation benchmark (SMRB) that provides a standard environment to enhance comparability between models and credibility of results. We implement eight models (five from top service computing conferences and journals and three created by ourselves) based on SMRB and compare their performance, which proves the effectiveness of SMRB. After analyzing these results, we found that most DL-based models do not perform as well as they promise; instead, the simplest Multilayer perceptron (MLP) models perform better after tuning the parameters, which inspires us to re-examine whether the particular structure of the model can be helpful for the intended purpose and whether it can really improve the performance of the recommendation.