Coconut: Multilevel Collaborative Deployment for Real-Time Deep Learning Tasks in Heterogeneous Edge GPU Cluster

With the rapid development of artificial intelligence, intelligent manufacturing factories usually deploy various deep learning models on some heterogeneous edge devices to process data or tasks generated by sensors. Scheduling these tasks in such a resource-constrained and heterogeneous edge cluster to satisfy their real-time requirements is challenging. Existing task deployment frameworks for edge clusters optimize at the model level, lacking fine-grained resource awareness and concurrency control, where the urgent tasks are frequently blocked and miss their deadlines. Therefore, we propose a multilevel collaborative deployment framework Coconut for real-time deep learning tasks in the typical heterogeneous edge GPU cluster. Coconut collaboratively optimizes model deployment and fine-grained concurrency control. To address the high complexity of multilevel collaborative optimization, we use an efficient learning-based search algorithm. Based on the operator-level information, we also pretrain an accurate latency predictor for each device, enabling centralized optimization to further accelerate the search. We implemented and validated Coconut on a real-world case of intelligent steel structure manufacturing. Compared with the advanced task deployment framework, Coconut improves the deadline satisfaction rate by more than 27% and the cluster GPU utilization by more than 18%. Coconut is an offline optimization framework with almost no additional runtime overhead.