Man-Machine Collaborative Task Planning Based on Visual Language Models

Traditional robot planning systems frequently exhibit limited semantic adaptability in unstructured environments and dynamic human-robot collaboration scenarios. To overcome this limitation, this paper proposes a hierarchical collaborative task planning framework based on visual language models (VLMs). Specifically, we use the Open X-Embodiment dataset to perform parameter efficient fine-tuning (PEFT) on the Open Flamingo model to align the visual language representation with the robot operation space. To enhance robustness against environmental and interaction uncertainties inherent in collaborative scenarios, we introduce a progressive react prompting (PRP) mechanism. This mechanism implements a structured three-level correction strategy, including local, sub-task, and global adjustments, to monitor the task execution status in real-time and correct errors. Empirical evaluations on the CALVIN benchmark show that our proposed method significantly outperforms the baseline method in terms of long-horizon task success rate. Additionally, practical experiments conducted on the Franka Emika arm verified the feasibility of this framework in complex collaborative environments.