Hi-RAGrasp: A Human-in-the-Loop Experience-Augmented Method for Task-Oriented Grasping

With the growing demand for assistive robots in aging societies, task-oriented grasping in household environments has become increasingly important. Compared with structured industrial settings, household scenarios are characterized by diverse objects, unstructured layouts, and strong variability in task semantics. However, traditional methods focus on geometric stability and fail to capture task-relevant semantic constraints on manipulation regions, while existing approaches suffer from unstable reasoning and lack effective mechanisms for incorporating human intervention into the reasoning process. To address these challenges, we propose Hi-RAGrasp, a task-oriented grasping framework that integrates progressive multi-stage reasoning, Human-in-the-Loop (HITL) interaction, and Retrieval-Augmented Generation (RAG). A coarse-to-fine pipeline progressively refines predictions from object-level localization to part-level grounding, enabling robust mapping from human instructions to fine-grained task-relevant regions. Meanwhile, a HITL correction mechanism and a structured human experience database are introduced and combined with RAG to form a unified paradigm that aligns with prior experience when available and falls back to reasoning otherwise, enabling experience reuse and future experience accumulation without retraining. In addition, a Geometric Heuristic Segmentation (GHS) method is proposed to improve task-relevant region localization for textureless objects. Experiments show that our method achieves a segmentation success rate of 77.73% on the evaluation dataset and a grasp success rate of 75% in real-world scenarios, significantly outperforming existing methods and demonstrating strong effectiveness and practicality in open environments.