generated
ArticulationDreamFAR: Real‑Time and Long‑Horizon Interaction with Diverse Objects
10+ FPSreal-time interaction
5+ minutesof autoregressive rollout
1000+ taskswith diverse objects
Try it live!
Mouse · Keyboard · Hand (webcam) · Leader arms
One visitor drives at a time in 4-minute sessions; while you wait you watch the current player live. Hand mode uses your webcam; frames are processed in memory and never stored.
Method
DreamFAR is a latent flow-matching world model: a tokenizer encodes each camera view into a compact latent, and a causal transformer predicts the next latent from the action, a window of recent latents and a few long-term memory frames. During rollout its own predictions become its history, so one network simulates recorded actions, a policy or a human operator.
1Unit-sphere tokenizer
Each camera view is encoded as per-position latents projected onto the unit sphere and decoded back to pixels by a flow decoder.
2Contact-based memory
When the robot makes or breaks contact with an object, DreamFAR inserts the corresponding keyframe into contact memory, indexed by end-effector pose.
3Streaming transformer
DreamFAR uses a causal transformer with a number of design decisions that make it well suited to streaming and long-horizon generation.
Results
Interaction with diverse objects
A single model produces stable AR rollouts across diverse objects, including deformable, compositional, and tool use scenes.
Comparison against baselines
The video shows a 60-s rollout on an episode of the “write a sentence on a whiteboard” task, DreamFAR against the existing baselines. DreamFAR is the only model that keeps the scene layout and the manipulated objects consistent over the whole minute and even predicts writing on the whiteboard that resembles the ground truth. The curves show metric drift on ABC over one minute. DreamFAR degrades the slowest on all three metrics and significantly outperforms all baselines.
0:00 / 1:00
time since hand-over (s), 10-s bins · cursor = rollout time of the video above
Policy evaluation inside the simulator
We run policy evaluation on both the real robot and inside DreamFAR. (a) Task progress over time for one episode of the “t-shirt folding” task, on the real robot (crosses) and inside DreamFAR (circles). (b) Mean progress per policy and task. The simulator is generally optimistic in absolute progress but preserves the ranking on two of three tasks.
Co-training with simulated demonstrations
DreamFAR enables real-time teleoperation without a real robot, so we collect 75 demonstrations of the “place bags of chips into a basket” task both on the real robot and inside the simulator. The π0 policy fine-tuned on the combination of both nearly doubles the real-only performance.
Ablation on memory mechanism
Contact-based memory keeps placed objects persistent. Three models with the same architecture but different memory selection: without memory the bin is empty, sparse time-strided memory hallucinates its contents, and contact-based memory matches the real observation.
