arxiv:2609.12541
Published on Sep 11
· Submitted by
JillJia on Sep 15
Upvote
17
Authors:
,
,
,
,
,
Abstract
A general-purpose agent directly controls a physical robot by interpreting visuals, writing executable programs, and revising actions based on physical feedback across diverse manipulation tasks.
Generated by thinkingmachines/Inkling-Small
We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. AGP achieves success rates of 100%, 100%, and 80% on three block construction configurations. These findings establish a path for general-purpose agents to act as robotic policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.
View arXiv page View PDF Project pageGitHub 10 Add to collection
Community
Paper submitter 8 days ago
Agent as Policy (AGP) connects a general purpose coding agent directly to a physical robot. The agent interprets camera observations, writes programs, requests motions, and adjusts its actions using physical feedback while model weights stay fixed. Evaluated tasks include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. AGP achieves at least 8 successful trials out of 10 for each evaluated configuration in assembly, block construction, and dice flipping. Reusing saved procedures and programs reduces execution time across repeated trials.
Paper submitter 8 days ago
•
edited 8 days ago
The agent trajectories are available on HF Mirror.
Paper submitter 8 days ago
A short video showing AGP on real robot tasks and how the agent controls the robot through tool calls and visual feedback.
7 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
HODAgent: Towards On-Demand, Responsive Humanoids for Physical World Human Interaction (2026)
-
A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution (2026)
-
Teach and Grow: An Agent-Centered Architecture for General Robot Learning (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
17
Get this paper in your agent:
hf papers read 2609.12541
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.12541 in a model README.md to link it from this page.
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.12541 in a Space README.md to link it from this page.