DeepFill: Accelerating MLLM Training by Filling Bubbles with Frozen Encoders

The training of multimodal large language models (MLLMs) has emerged as a crucial area in artificial intelligence, aiming to integrate diverse modalities such as text and images into a unified framework. Due to the vast number of model parameters, MLLM training often employs techniques such as pipeline parallelism (PP) and the Zero Redundancy Optimizer (ZeRO) to address memory limitations. However, existing frameworks, such as DeepSpeed, fail to fully utilize GPU resources, resulting in significant idle time when distributing the training task across multiple devices. To mitigate this issue, we introduce DeepFill, a novel framework designed to enhance MLLM training efficiency. First, DeepFill separates the inference component of the frozen modality encoders from the training process of the main large language model (LLM), assigning them to distinct execution streams. Second, DeepFill leverages the idle GPU time in PP and ZeRO and precomputes the encoders within a single training step (PP bubbles) or between two consecutive steps (ZeRO bubbles). Our experiments on two open-source MLLMs demonstrate that DeepFill significantly improves the training throughput by up to 1.08× for PP, 1.11× for ZeRO and 1.18× for their hybrid optimization, closely aligning with theoretical expectations.