Multitask Cooperative Genetic Programming for Co-Scheduling Online-Offline Workflows in the Cloud

The underutilization of cloud resources remains a significant challenge due to overprovisioning and resource silos. While co-scheduling latency-critical online workflows with best-effort offline workflows is a promising strategy, existing approaches often overlook the complex bidirectional coupling and fine-grained resource dependencies between these workloads. This paper studies the Co-Scheduling of Online-Offline Work-flows problem, formulated as a coupled multitask optimization model that jointly accounts for fine-grained CPU/memory configuration and parallel task execution within virtual machines. The objective is to simultaneously minimize the total flowtime of online workflows and the total cost of shared cloud clusters. We propose a Multitask Cooperative Genetic Programming (MCGP) approach to address the dynamic interaction where the scheduling decision of one task alters the environment of another. MCGP automatically evolves and learns two dedicated rule pairs, comprising task and resource selection rules, tailored for online and offline workflows, respectively. MCGP enables the two scheduling strategies to collaborate effectively within a shared environment by integrating multitask learning for knowledge transfer and cooperative coevolution for strategy co-adaptation. Extensive simulation experiments based on real-world traces show that MCGP consistently outperforms existing baselines in terms of reducing total flowtime and costs, and improving the success rate.