Project / 02 · 2026开源项目 / 02 · 2026

RealMan AgentEmbodied agent

Embodied ReAct & Dual-Layer Memory for Long-Horizon Mobile Manipulation

An embodied ReAct agent framework with Working Memory and SQLite Episodic Memory for long-horizon mobile manipulation, connecting navigation, YOLO-seg + FoundationPose 6D pose estimation, and grasping on RealMan hardware.

RealMan Agent editorial illustrationFIG. 02 / EDITORIAL STUDY
ROLEFramework architecture & mobile manipulation integration
STATUSReal-robot system

Research question核心研究问题

How can a mobile manipulator orchestrate multi-step navigation, perception, and grasping from natural language while turning past failures into reusable planning constraints?

Why it matters设计视角

This page treats the project as a system and a research record, not as a list of technologies.将工程项目视为完整的实验记录与系统架构,而非孤立技术的堆叠。

System系统流水线

01Instruction + scene graph02Working Memory state03SQLite Episodic Memory04ReAct Tool Calling05Chassis navigate_to06YOLO-seg + FoundationPose 6D07Arm grasp + Verifier

Method & contribution核心方法与贡献

Built the ReAct closed-loop planner paired with Working Memory (live mission/robot state) and SQLite Episodic Memory (failure reflection and cross-task lesson retrieval via Qwen), integrating chassis navigation, YOLO-seg segmentation, FoundationPose 6D pose estimation, and Verifier feedback.

PERSONAL RESPONSIBILITY / 个人负责
  • 设计面向长时序移动操作的 ReAct 闭环决策器与 Tool Calling 技能编排
  • 构建 Working Memory 实时状态维护与 SQLite Episodic Memory 失败归因反思机制
  • 集成底盘导航、RGB-D + YOLO-seg 分割、FoundationPose 6D 位姿估计与 RealMan 机械臂抓取
  • Ground each ReAct decision in real-time robot feedback rather than open-loop scripts
  • Keep preconditions explicit in Working Memory to prevent blind repeated retries
  • Consolidate failure trajectories into conditional planning constraints stored in SQLite
  • Separate high-level agent orchestration from low-level vision and ROS driver adapters

Evidence实验证据与交付物

OUTCOMES / 当前成果
  • Validated A→B→A object retrieval pipeline with offline replay and Qwen API reflection benchmarks
  • Documented ROS packages, FoundationPose adaptation, and hardware validation interfaces
ARTIFACTS / 可核验证据
  • End-to-end A→B→A eyeglasses-case retrieval scenario and reflection benchmark
  • Hardware validation guide and backend API documentation
  • Reproducible offline memory and reflection demonstration scripts
Limitations / failure record已知局限与探索边界

Long-horizon reliability depends on accurate 6D pose transforms, precondition verification fidelity, and Qwen reflection prompt quality under novel physical disturbances.