Learn VLA, imitation learning, camera calibration, and Sim2Real through the tidy-up robot "The Tidy-Up Ghost" ("Shimatte-AI").
After AI that generates text and images, what the world is now grappling with is AI that sees, thinks, and physically moves ——— Physical AI. NVIDIA's Jensen Huang said "every industrial company will become a robotics company." It may sound like an overstatement. But for Amazon, this is not a prophecy about the future—it is a daily reality.
Over the past year or two, AI has been shifting from "creating text and images inside a screen" toward "seeing reality, thinking, and moving a body." What supports this is the emergence of foundation models such as VLA (Vision-Language-Action)—the star of this book—and world models that predict how the world behaves. "Give an instruction in words, and the robot moves even in an environment it has never seen"—that kind of generalization is becoming real. This is why people say a "ChatGPT moment for robotics" is near.
It is surprisingly little known, but Amazon is one of the companies that moves the physical world at the largest scale on Earth. Since welcoming a warehouse-robotics company in 2012, it has developed in-house autonomous mobile robots, item picking, and even a robot with a "sense of touch" (Vulcan), and in 2025 its deployed robots surpassed a cumulative one million units (across more than 300 fulfillment sites).
And at the same time, it announced DeepFleet ——— a "generative AI foundation model for robots" that learns the movements of hundreds of thousands of robots and coordinates the entire fleet. It also invests in outside robotics and humanoid companies. In other words, for Amazon, Physical AI is not a research theme but a backbone directly tied to delivery speed and cost. It has bumped into—and solved—the difficulty of "running the real world with AI" every single day, and that know-how becomes the foundation for the next generation of AWS.
Physical AI is a massive new workload for the cloud. Huge amounts of data, heavy simulation, GPU training, real-time inference at the edge, and operating many-unit fleets—the work it generates maps directly onto AWS's strong areas. AWS has built dedicated information channels and reference architectures for Physical AI and collaborates deeply on GPU simulation infrastructure because it positions this as a first-class theme.
Physical AI consists of a long loop: "data collection → simulation → training → Sim2Real → edge hardware → fleet operation." AWS's strength is that you can run this entire loop on a single foundation.
The "Tidy-Up Ghost" in front of you is proof that this end-to-end flow is actually running—not just on slides. With an arm of about ¥40,000 (plus a separate inference Jetson Thor in the ¥500,000 range) × open foundation models × AWS, you can build a Physical AI that tidies up a room when you speak to it. This book unpacks what is inside, all in one volume. The next robot you bring to life can start right here. *shiver… shiver…*
A small robot arm (SO-101) sits in a miniature room. When a visitor speaks into a microphone—"put away the eraser"—the AI understands the words, moves the arm, and tidies up the scattered objects. The brain that drives the robot is the VLA model; teaching that VLA "how to move" is imitation learning; accurately measuring how the camera sees is camera calibration (ChArUco); and practicing in a virtual space instead of on real hardware is Sim2Real. This book is built around these four pillars, adding the overview (§1) and edge/cloud design (§6).
For decades, industrial robots in factories were programmed move by move by humans. "Joint 1 to 30 degrees, joint 2 to −15 degrees, then grip"—coordinates and angles written in directly (teaching). This is accurate for handling fixed objects at fixed positions, but it has the following weaknesses.
In contrast, recent Physical AI aims to see with a camera, judge for itself, and move. Instead of a human specifying coordinates one by one, the robot itself decides how to move its joints from the verbal instruction "put away the eraser" and the camera images. The thing that handles this "see → judge → move" is the star of this book, the VLA model.
A robot's motion runs in a loop, just like human reflexes.
The key is not "output the correct move all at once," but to repeat move a little, look, move a little more at high speed. That is why the speed of this loop (how small the latency is) is decisively important. In Chapter 6, "Edge vs Cloud," we explain why this loop must run on the computer at hand (the edge).
The division of roles between "smart but somewhat slow thinking (cloud)" and "fast reflexes (edge)" is the backbone of this system.
First, a review. Recent AI includes the VLM (Vision-Language Model). This is a model that can handle "images" and "language" at the same time—for example, show it a photo and ask "what is this?" and it answers "a cat." It is an image-understanding + language model (think of GPT-4V or Claude's image features).
A VLA (Vision-Language-Action) adds an "action" output to this VLM.
VLM: image + language ──→ language (description / answer) VLA: image + language + robot state ──→ action (how to move the joints next)
In other words, a VLA is a model that "sees, understands the verbal instruction, and outputs how to actually move its body." It is the core technology of Physical AI.
Using the SO-101 arm that the Tidy-Up Ghost uses as an example, let us make the VLA's inputs and outputs concrete.
A VLA predicts not step by step, but predicts future actions together as a chunk. This is called Action Chunking. The reason is to avoid the compounding error.
If you predict step by step, even a tiny prediction error makes the next input that shifted state, which produces an even more shifted prediction, and so on—the error snowballs. On the other hand, if you output a whole chunk of actions at once, even if there is some wobble along the way, it is easier to keep the overall trajectory smooth. This idea was popularized by the research ACT (described later).
A language model picks "the next word" probabilistically, but a robot's joint angles are continuous values (30.5 degrees, 31.2 degrees…) and must connect smoothly. To generate this continuous action sequence, recent VLAs use methods called flow matching and diffusion.
Intuitively, it does the same thing that an image-generation AI (such as Stable Diffusion) does—gradually removing noise from a "noisy image" to create a clean picture—but applied to an "action sequence."
random noisy action sequence ──(a few steps of "denoising")──→ smooth, meaningful action sequence
▲
the image and language instruction condition "what the action should be"
Picture "carving a clean motion out of noise." π0.5, SmolVLA, and GR00T all adopt methods of this lineage.
In VLA design, a two-layer structure modeled on human cognition often appears (NVIDIA GR00T makes it especially explicit). The phrasing "System 1 / System 2" was popularized by psychologist Daniel Kahneman's bestseller Thinking, Fast and Slow.
This idea of separating a "smart but slow layer" from a "fast layer" maps neatly onto Chapter 6's division of roles between the cloud (smart thinking) and the edge (fast reflexes). Keep it in mind.
In the Tidy-Up Ghost project, multiple VLAs were compared in training and validated on real hardware. We trained π0.5, ACT, GR00T, and SmolVLA, and the live demo at the production booth ran ACT on three tasks (eraser, marker, die). Here we organize the characteristics of the four representative models.
| Model | Developer | Scale | Language instruction | Action generation | Characteristics | Role in this demo |
|---|---|---|---|---|---|---|
| π0.5 | Physical Intelligence | large (a few B) | yes | flow matching | strongest generalization · web-data co-train | validation: 3 tasks |
| GR00T N1.5–N1.7 | NVIDIA | large (3B) | yes | DiT/flow | System1/2 two layers · human video/synthetic | hardware validation: picked the red pen |
| SmolVLA | Hugging Face | small (~450M) | yes | flow matching | lightweight · async inference · edge-capable | training · validation |
| ACT | Stanford(ALOHA) | small | no | CVAE+Transformer | lightweight · task-specific · birthplace of chunking | production live: 3 tasks |
We train and compare π0.5 / ACT / GR00T / SmolVLA in the cloud (SageMaker), searching while recording in MLflow "which one tidies up best." Training data uses two streams: teleoperation (a human demonstrating on real hardware) and Sim (synthesized in a virtual space). The trained models are deployed to the edge for inference and validation on real hardware (delivered via Systems Manager + S3; Greengrass auto-delivery is a future option). "The field uses something reliable; research explores broadly, including large models"—we run this cycle to compare, and for this booth we adopted ACT for the production live demo.
The Tidy-Up Ghost uses GR00T by FT-ing from one of the N1.5–N1.7 bases (the one that first reached picking the red pen on real hardware was N1.7). All three generations share the skeleton of "3B parameters, two layers of System 2 (VLM = thinking) + System 1 (action generation = reflexes)," but each generation evolves in its VLM head, action-generation scale, and training data.
| Generation | VLM (thinking) | Action generation (reflexes) | Highlight of training data | In a word |
|---|---|---|---|---|
| N1.5 | Eagle 2.5 (frozen) | 16-layer DiT + 4-layer adapter / absolute joint angles | GR00T-Dreams (synthetic) | first stable version |
| N1.6 | Cosmos-Reason-2B (variable resolution) | 32-layer DiT (2×) / adapter removed, top 4 VLM layers trained / relative actions | thousands of teleop hours added (bimanual, walking) | improved accuracy for two hands / walking |
| N1.7 | Cosmos-Reason-2B (enhanced reasoning) | Action Cascade (separates reasoning and control) | EgoScale = ~20,000 hours of human first-person video | reasoning-enhanced "think then act" version |
The training scripts are built to FT from nvidia/GR00T-N1.5-3B / N1.6-3B as the base, and the one that first reached picking the red pen on real hardware was N1.7. The higher the generation, the more it tends to be "strong on unseen objects and procedures even with few demonstrations." After training and hardware-validating three models, we adopted ACT for this booth's production live demo (π0.5 and GR00T were also validated on hardware).
This is a reading column slightly off the main path. If you are in a hurry, you can jump to §3 Imitation Learning and still follow the main thread (the relationship between these four models and the world's major VLAs is mapped out in §9 Appendix).
The π0.5, GR00T, SmolVLA, and ACT we have met so far differ in their makers, their philosophies, and "how much they have made public"—and those differences are themselves a microcosm of the VLA industry. Let us take a sweeping look at each model's background, history, and trivia, along with how open its code / weights / training data are.
tonyzhaozh/act). However, ACT is an "algorithm" rather than a model—there are no huge pre-trained base weights like π0/GR00T, and you train it each time on your own demonstrations (so the notion of "published weights" is thin; you collect the data yourself too).| Model | Maker | Code | Weights | Training data | License |
|---|---|---|---|---|---|
| π0 / π0-FAST / π0.5 | Physical Intelligence | ✅ public | ✅ public | ❌ not public | Apache 2.0 |
| π0.6 / π0.7 | Physical Intelligence | ❌ | ❌ | ❌ | paper only (weights closed) |
| GR00T N1–N1.7 | NVIDIA | ✅ public | ✅ public | △ partially public | code Apache 2.0 / weights NVIDIA Open Model License |
| SmolVLA | Hugging Face | ✅ public | ✅ public | ✅ public (community) | Apache 2.0 |
| ACT | Stanford (ALOHA) | ✅ public | – (trained each time) | – (self-collected) | MIT |
The π0.5, GR00T, and ACT this demo runs on real hardware, plus the SmolVLA initially validated, are all openly obtainable (the only closed ones are the latest π0.6 / 0.7). This demo's claim that "you can build an enterprise-grade demo with a ¥40,000 arm × open foundation models" is backed by this fact (the inference edge computer Jetson Thor is separate, in the ¥500,000 range).
Moreover, the four models have "kinship." The lineage runs ACT (Stanford) → its authors Levine and Finn founded Physical Intelligence → π0. And flow matching (generating continuous motion) and Action Chunking (predicting actions as chunks) have become DNA shared by nearly all the models.
Lining up openness in a row—SmolVLA (code + weights + data, all) > π0.5 and GR00T (code + weights) > ACT (code only = a method) > π0.6 / 0.7 (paper only). There is a tendency for the cutting edge to hold weights close, but what is interesting is that robot foundation models have an even more deeply rooted "open culture" than LLMs.
In natural language processing (NLP), every time a new invention appeared, "generality" leapt up in stages. There was word2vec, which turned words into meaning vectors; the Transformer, a general-purpose architecture that captures context as a whole; BERT, which pre-trains in advance on large amounts of text; and GPT, which pushed that toward generation and enormous scale. With GPT-3 in particular, a single model could handle translation, summarization, question answering, and more just by "being shown an instruction or examples," without rebuilding the model. This "turning point at which task-specific designs become unnecessary and a general-purpose model can do anything from instructions alone" is metaphorically called the "GPT moment" (the same phrasing as the "ImageNet moment" that refers to 2012, when deep learning transformed image recognition).
This frame partly applies to VLA too. The Transformer and pre-trained vision-language models (VLMs) are borrowed wholesale by VLA—that is, VLA has already acquired the foundation corresponding to word2vec and the Transformer. In fact, π0.5 and GR00T are entering the stage of "handling multiple tasks if instructed in words," which can be called a small "sign of GPT."
So why is it said that "VLA's GPT moment has not arrived yet"? NLP's true fuel was not the Transformer alone, but internet-scale text (nearly free, learnable in a self-supervised way). Yet robot action data is not lying around on the net—it can only be collected by hand via teleoperation (this "scarcity of data" is detailed in the column at the start of the next section, §3). There is infinite video, but it does not come with the answer of "which joint was moved and how"—and the absence of a mechanism to "convert abundant video into a teaching signal a robot can use" is said to be the biggest wall right now.
What is therefore seen as promising is a data engine that learns at scale from human video and virtual spaces (simulation) (GR00T N1.7's EgoScale is one example), and a World Model that acquires physical common sense by predicting "what the world will become next." The latter is drawing attention with the idea that "VLA's next GPT might be the world model." In summary—the architectural "GPT moment" is mostly achieved for VLA too. What remains is a breakthrough in "data" and "world-model-style learning," and that is "coming, but not yet fully here"—that is where we stand in 2026.
The "brain" called a VLA does not know how to tidy up from the start. It learns by mimicking human examples. This is called imitation learning, and the most basic method in particular is called behavior cloning.
"An LLM can do all sorts of things from just an instruction (a prompt), so why does a VLA need extra training?"—a reasonable question. But the two are not different mechanisms. The reason an LLM can work "from just a prompt" is the result of pre-training on internet-scale text; the prompt merely calls up that knowledge. A VLA, too, is instructed in words at inference time—"put the marker in the basket" (= equivalent to a prompt). The point that one model handles multiple tasks without retraining per command is the same as well.
The difference comes from two asymmetries. ① Scarcity of data: text is virtually infinite on the net, but "robot action data" can only be collected by hand via teleoperation and is orders of magnitude scarcer ("internet-scale robot motion" does not exist). ② Grounding in a body (embodiment): an LLM's output is the same "token" everywhere, but a VLA's output is a motor command tied to this SO-101, this camera, this physics. Even the same "tidy up" has to be fitted to one's own body. So the process of "practicing on this body, in this field"—fine-tuning—is still needed for now.
By analogy, an LLM is a person who got smart by reading a huge number of books (the reading material is infinite). A VLA is a person who cannot use a new tool without practicing with their own hands; reading the manual alone will not move their fingers. Fine-tuning corresponds to that "practice with one's own body." Note that this gap is shrinking—π0.5 aims for generalization to "houses it has never seen," and GR00T aims for transfer between robots with minimal extra training—VLA is at the stage of following the path the LLM trod.
The basics of machine learning, "supervised learning," show a large number of input-and-answer pairs to learn the input→answer correspondence. Example: show many "photos + correct labels (cat/dog)" so it can guess the animal in a new photo. Imitation learning applies this to robot actions:
input (observation) = camera image + language instruction + current joint angles answer (teacher) = the action the human took at that moment (how the joints were moved)
You collect a large number of these pairs to learn "in this situation, move like this."
So how do we record "the correct action the human took"? This is where teleoperation (remote operation) comes in. The SO-101 has two kinds of arms.
When a human moves the leader arm to demonstrate "put the eraser in the basket," the follower performs the same motion. At this time, three things are recorded at each time step:
one frame at time t: ├─ camera images (overhead + wrist) ← observation ├─ joint angles at that moment ← observation (self-state) └─ joint angles the human moved to next (= action) ← teaching signal (answer)
Lining up these frames from the start to the end of the motion gives one episode (one example run). You repeat this tens to hundreds of times to build a dataset. The Tidy-Up Ghost accumulates it in the dataset format of LeRobot (Hugging Face's robot-learning library).
Training a VLA, in the end, is nothing other than approximating with a neural network the "correspondence function from observation space → action space (called a policy)."
Roughly, training a VLA that uses flow matching goes like this:
Repeat this millions of times, and the model learns "given this observation, here is how to carve the action out of noise." At inference time, it starts from pure noise and generates a smooth action by the learned procedure (§2.4).
In production (inference), the model has no correct action at hand—only the image and the instruction. So it has no choice but to create the action from "blank noise" (§2.4). But guessing the finished form straight from noise is hard, so during training we make a large number of samples that break the answer at various strengths and have it learn "an arrow that steps one move back toward the answer from any degree of brokenness." In production it follows that arrow over and over to carve the action out of noise.
By analogy, think of sculpting. Someone who knows the finished form repeatedly practices "where to carve next to get closer to completion" on stones deliberately roughened in various ways, and in production carves a statue from a single block of stone with the learned hand movements—that is the image.
This is the heart of the modern VLA. π0.5 and SmolVLA are distributed as pre-trained foundation models. They are already in a state where they have learned general-purpose body movements—"grasp, carry, place"—from vast robot data around the world.
What the Tidy-Up Ghost does is to additionally train this foundation model a little, to fit its own task, camera placement, and arm. This is called fine-tuning (FT).
| Item | Value | Note |
|---|---|---|
| Model | π0.5 / ACT / SmolVLA / GR00T trained and hardware-validated | The production live demo used ACT |
| Number of episodes | 20–300 | e.g., 300 episodes at ≈30 s each is about 3 hours of collection |
| Number of epochs | 50 | passes over the whole dataset 50 times |
| Training machine | g6.2xlarge / g6e.2xlarge | AWS GPUs. About 6 hours for 20 episodes |
| Experiment management | SageMaker + MLflow | records and compares which settings have the highest success rate |
The lighting at the actual venue is always different from when the data was collected. So that the model does not malfunction with lighting changes, we deliberately collect data while varying the lighting patterns (partial lighting, full lighting, full darkness, etc.). In this way it acquires the robustness to "move the same way even when the appearance changes somewhat." This idea of "deliberately injecting variation into the training data" is the same spirit as domain randomization in the next chapter, Sim2Real.
When you hear "robot learning," you might picture reinforcement learning (RL), where the robot maximizes a reward through trial and error. But the VLAs that serve as Shimatte-AI's brain (π0.5, GR00T, ACT, SmolVLA) are all imitation-learning-based. Why not RL?
On the other hand, for locomotion and balance control (standing and running on legs), the reward is easy to specify and simulation is fast, so RL is still mainstream. In other words there is a division of labor: "manipulation that picks things up → imitation-learning side / walking and balance → reinforcement-learning side." Shimatte-AI is arm manipulation, so it sits on the imitation-learning side.
Recently the combination of "build the base with imitation learning, then polish with reinforcement learning" is also growing (similar in spirit to the "pretraining + RLHF" of large language models). Shimatte-AI likewise keeps reinforcement learning in view as room for future extension.
A VLA moves by relying on camera images. But a camera is not an "honest eye." Its lens has distortion, and the same object is captured differently depending on the model and settings. The work of accurately measuring these "camera quirks" is camera calibration, and its standard tool is the ChArUco board.
If the way the camera used to train the VLA captures images differs from the camera used for inference, the training goes to waste. Furthermore, in Chapter 5's Sim2Real, the "virtual camera in simulation" must be made to capture images the same way as the "real camera." To do that, we must know in numbers "what quirks the real camera has." Calibration is the work of finding these numbers.
A camera is mathematically approximated as a pinhole camera. The correspondence that determines where a 3D point lands on the 2D image after passing through the lens is the intrinsic parameters (intrinsics). The intrinsics consist of two parts: ① the camera matrix K and ② the distortion coefficients (the distortion coefficients are a separate part, not part of K's contents).
① Camera matrix K … a 3×3 matrix representing the camera's basic performance. Its contents are the following two kinds:
② Distortion coefficients … separate from K. Coefficients that correct lens distortion (e.g., straight lines bending into a barrel shape with a wide-angle lens).
3D point (X, Y, Z)
│ passes through the lens (pinhole)
▼
pixel on the image (u, v) ← K and the distortion coefficients determine this correspondence
┌ fx 0 cx ┐
│ 0 fy cy │ ← this is the camera matrix K (3×3)
└ 0 0 1 ┘ distortion coefficients (k1,k2,p1,p2,…) are held separately, outside K
The purpose of calibration is to find this K and the distortion coefficients by measurement.
Traditionally, a chessboard (checkerboard pattern) is used. The corners of the black-and-white grid can be detected with very high precision by image processing. You photograph a grid of known size from various angles, gather many correspondences between "the corner's position in the real world" and "the corner's position on the image," and back-calculate K and distortion.
Weakness: if part of the board goes outside the frame or is hidden by a hand, you can no longer tell which corner is which, and calibration fails. This is because all the chessboard corners look alike and cannot be distinguished.
An ArUco marker is a black-and-white square marker, like a simplified QR code. Each has a unique ID (number) and has the following strengths.
Its weakness is that the corner position accuracy is not as high as a chessboard.
Trivia: the name ArUco comes from "Augmented Reality University of Córdoba" (the University of Córdoba in Spain, its developer). It is a bona fide academic tool proposed in a 2014 paper.
ChArUco is, as the name suggests, Chessboard + ArUco. It is a board with ArUco markers embedded in the white squares of a chessboard grid.
┌─┬─┬─┬─┐ │ │▣│ │▣│ ▣ = ArUco marker (has an ID → strong at partial detection) ├─┼─┼─┼─┤ ＋ = chessboard corner (high precision) │▣│ │▣│ │ ├─┼─┼─┼─┤ │ │▣│ │▣│ └─┴─┴─┴─┘
This combines the strengths of both:
As a result, you can calibrate stably and with high precision even when the board is slightly hidden by a hand or arm, or at an angle. It is well suited to calibration in a "place where things move around," such as a robot's workspace.
intrinsics.yaml).The key is that the board's dimensions, squares, and marker IDs are "exactly known." In other words, the board is a ruler with known graduations. With just one shot, the answer is not determined (e.g., you cannot tell "a large board far away" from "a small board up close" = insufficient clues). But take dozens of shots while varying the angle and distance, and the only thing that changes each time is the board's position and orientation, while the camera's own quirks (K and distortion) are common to all shots. Because you narrow down to one set of "K and distortion that can explain all the photos at once" by optimization, it is found precisely. In short, it is a matter of "one shot = too few clues, many shots = an over-determined system you solve simultaneously."
From the contents of the calibration repository camera_calibration, here are this demo's concrete values:
| Item | Value |
|---|---|
| ChArUco board | A4 size, 7×10 squares, 26.0 mm per square |
| Overhead camera (OBR-WEBCAM200) | field of view (FOV) about 49.9° |
| Wrist camera (InnoMaker U20CAM-1080P) | field of view (FOV) about 72.5° |
| Output 1 | intrinsics.yaml (camera matrix K + distortion coefficients) |
| Output 2 | isaac_camera.yaml (camera settings for the simulator Isaac Sim) |
The overhead camera sees the whole room, so its field of view is kept narrower (49.9°), while the wrist camera sees nearby areas widely, so it is wide-angle (72.5°). This second output, isaac_camera.yaml, is the bridge to the next chapter, Sim2Real—a file for transplanting the "real camera's quirks" measured in calibration directly onto the simulator's virtual camera.
Note: this demo has only these two cameras. The wrist camera is the "eye" attached to the wrist of the SO-101 arm, and the overhead camera is fixed at a distance to see the whole room. The SO-101 does not have a separate built-in camera (the same two cameras as "one overhead + one wrist" in §2.2).
As we saw in Chapter 3, training a VLA requires a large amount of demonstration data. But collecting data on real hardware is hard:
This is where simulation (a virtual space) comes in. If you recreate a room and a robot inside a computer, you gain advantages like fast and parallel (you can run hundreds of virtual robots at once), safe and unlimited (if it breaks, just restart), and the exact answer is known (you obtain the precise positions of objects from a "god's-eye view").
Doing this—"learn in simulation → run on real hardware"—is called Sim2Real (Simulation to Reality). The Tidy-Up Ghost uses NVIDIA's Isaac Sim (a robot simulator with physical simulation + photorealistic rendering).
Trivia: NVIDIA's robot foundation brand "Isaac" is named after Isaac Asimov, the SF writer who gave us the Three Laws of Robotics. This book's Isaac Sim, and GR00T's formal name "Isaac GR00T," are both part of this Isaac family (= the Asimov mark) (note: the edge computer Jetson is a separate brand).
In general the image of "simulation = where you do reinforcement learning" is strong (training walking robots, for example, really is running RL inside the Sim). But Shimatte-AI's Sim is used not as a place for reinforcement learning, but as a data factory for imitation learning.
The key point is that in the Sim you can produce demonstration data without a human operating anything. In the virtual space you can obtain the exact position of objects from a "god's-eye view," and generate optimal trajectories automatically, in parallel, and faster than reality at large scale. This is mixed into the real-hardware teleoperation data for training (the synthetic data generation of §5.4).
In short, Shimatte-AI's Sim has three roles: (1) automatic generation of demonstration data, (2) accumulating data faster by running in parallel with real-hardware collection, and (3) safe pre-validation before trying on real hardware. None of these is "trial and error driven by reward (RL)"; they are all ways to support imitation learning. That is why this demo achieves Sim2Real without using reinforcement learning.
The core problem of Sim2Real is the domain gap: because the virtual space and reality never match perfectly, a model that worked well in simulation fails on real hardware. The gap is mainly of three kinds:
Because a VLA moves by looking at camera images, the "visual gap" is especially fatal. If the training images (Sim) and the inference images (real hardware) are captured differently, the hard-won training is useless.
In the Tidy-Up Ghost, we recreate the same SO-101 arm as the real hardware inside Isaac Sim, and match the geometry by swapping the standard arm's wrist camera, mount, and gripper for the same ones as the real hardware. And the decisive part is applying Chapter 4's calibration results (isaac_camera.yaml) to the virtual camera.
calibrate the real camera with ChArUco
│ measure K, distortion, FOV (49.9°/72.5°)
▼
isaac_camera.yaml
│ set those numbers on the virtual camera
▼
Isaac Sim's virtual camera captures images the same "way" as the real hardware
└─→ the visual gap shrinks → a VLA learned in Sim is more likely to work on real hardware
This is the answer to "why camera calibration is the key to Sim2Real." If you do not match the field of view and distortion, the Sim training images and the real-hardware inference images become systematically misaligned.
The other classic strategy. In simulation, deliberately scatter conditions at random: train while making lighting color and intensity, object texture and color, background, friction, and mass different every time. Then it accumulates the experience that "the colors and light differ every time, but in each case I could put the eraser in the basket," and the model learns the essential motion that does not depend on fine details of appearance. As a result, it adapts more easily to unknown reality (which the model regards as just one more pattern of the randomization). This is the same spirit as collecting data with varied lighting in §3.7.
We have reached the stage of building an SO-101 environment equivalent to the real hardware on Isaac Sim and matching the cameras with calibration values. In the production demo, we plan to show on the training dashboard that "we also train in Sim." Sim2Real is positioned to be expanded further going forward.
Finally, the most practically important point in system design. We organize "should this computation be done at hand (edge), or sent to the cloud" using the Tidy-Up Ghost as an example. This connects directly to the "loop speed is everything" story in §1.2 at the start.
The edge is close and fast but weak; the cloud is distant and has latency but is powerful. This contrast is the starting point of the design.
The Tidy-Up Ghost involves two inferences with completely different natures. Separating these is the biggest knack.
| ① Real-time control loop | ② Post-action success judgment | |
|---|---|---|
| What it does | outputs joint angles from camera images dozens of times per second | evaluates "is it tidied?" from an image just once |
| In charge | VLA model | VLM (semantic judgment of the image) |
| Latency tolerance | extremely strict a delay breaks control | loose 1–2 s is OK |
| Conclusion: where to place | must be the edge | the cloud is fine |
The control loop (camera → VLA → joint angles) needs to close dozens of times per second. If you send this to the cloud:
So VLA inference is run locally on the Jetson Thor. Running π0.5, ACT, and GR00T at the edge is precisely to "spin the loop fast at the edge" (§2.6). This high-frequency control is exactly the System 1 (fast reflexes) side in the System 1/2 analogy (§2.5).
On the other hand, the judgment "did the eraser go into the basket?" is a one-shot affair after the motion ends, and can wait 1–2 seconds. In fact, there are positive reasons to put it in the cloud:
This "smart but not urgent judgment" corresponds to the System 2 (smart thinking) side in the System 1/2 analogy (the precise correspondence is supplemented in §6.6).
| Judge at the edge | Judge in the cloud (adopted approach) | |
|---|---|---|
| Latency | minimal | 1–2 s (within tolerance) |
| Smartness of judgment | disadvantaged—only small models fit | advantaged with a large VLM |
| Network dependence | none (works offline) | yes |
| Extra cost | consumes the Jetson's VRAM | API call cost |
| Simplicity of implementation | complex—the VLA and judge coexist | simple—coexists with the conversational AI |
fast reflexes (high frequency, low latency is everything) → edge (Jetson) … VLA control loop = System 1 smart thinking (occasional, smartness is everything) → cloud (Bedrock) … VLM success judgment / conversation = System 2
The two-layer System 1 / System 2 structure of the VLA touched on at the start (§2.5) forms a close analogue of the edge/cloud division of roles. But this is only a metaphor—the System 2 (VLM = thinking) inside GR00T is part of the VLA running at the edge, and is a different thing from the success-judgment VLM we place in the cloud here. Take it to mean that the design key "fast reflexes at the edge, smart thinking in the cloud" is easier to see when likened to System 1/2.
As an intuition about the control loop, that is correct, and indeed VLA control runs at the edge. But limited to "one-shot success judgment," for the reasons above a cloud VLM is appropriate—that is this demo's design decision.
① Control loop = implemented at the edge. ② Success judgment = planned on a cloud premise but not yet coded (voice → Claude conversation → MQTT → edge works). For details, see the system overview document.
| Term | Reading / full name | Rough meaning |
|---|---|---|
| Physical AI | Physical AI | An umbrella term for the field of AI that perceives reality through cameras and the like and moves a body such as a robot. |
| VLA | Vision-Language-Action | A model that takes an image + a language instruction + self-state and outputs an action (joint angles). The star of this book. |
| VLM | Vision-Language Model | A model that handles images and language (does not output actions). Used as the "brain" part of a VLA. |
| Imitation learning | Imitation Learning | A framework for learning a policy by mimicking human examples. |
| Behavior cloning | Behavior Cloning | The most basic form of imitation learning. Supervised learning of "observation → human action." |
| Teleop | Teleoperation | A person operates the leader arm, and that motion is recorded as the correct-answer data. |
| Episode | episode | The data for one example run (from start to finish). |
| Observation space | observation space | The set of inputs to the model (images, joint angles, language instruction, etc.). |
| Action space | action space | The set of actions the model outputs (target angle of each joint, etc.). |
| Policy | policy | The observation → action correspondence function. What a VLA approximates. |
| Action Chunking | Action Chunking | A method that predicts future actions as a chunk rather than one step at a time. Prevents error accumulation. |
| flow matching | flow matching | A method that generates smooth continuous actions from noise. A relative of the diffusion model. |
| diffusion / DiT | diffusion model / Diffusion Transformer | A method that generates the target output by repeated denoising / its Transformer implementation. |
| Fine-tuning | fine-tuning | Additionally training a pre-trained model on a small amount of data for one's own task. |
| cross-embodiment | cross-embodiment | The property of being able to load a single foundation model onto multiple robots of different shapes (humanoid, arm, etc.). GR00T is originally for humanoids, but this lets it be repurposed for this demo's SO-101 arm. |
| Epoch | epoch | One pass over the entire training dataset. 50 epochs = 50 passes. |
| System 1 / 2 | — | Two layers of fast reflexes (1) and slow thinking (2). A common motif in VLA and edge/cloud design. |
| Camera calibration | camera calibration | Measuring a camera's intrinsic parameters (focal length, principal point, distortion). |
| intrinsics | intrinsic parameters | Focal length fx,fy / principal point cx,cy / distortion coefficients. Determine the 3D point → image pixel correspondence. |
| ChArUco | ChArUco | A calibration board combining a chessboard (high precision) + ArUco (IDs, partial detection). |
| ArUco | ArUco | A black-and-white square marker with an ID. Can be detected and identified even when partly hidden. |
| Sim2Real | Sim2Real | The technique of learning in simulation and running on real hardware. |
| Domain gap | domain gap | The difference between Sim and reality (appearance, physics, latency). The biggest hard part of Sim2Real. |
| Domain randomization | — | A method that randomly varies conditions in Sim to learn motion independent of appearance. |
| Isaac Sim | Isaac Sim | NVIDIA's robot simulator (physics + photorealistic rendering). |
| Edge | edge | The on-site computer at hand. In this demo, the Jetson AGX Thor. |
| OTA | Over-The-Air | Delivering and updating a trained model to edge devices over the network. |
| MQTT | — | A lightweight messaging protocol. Used for command delivery between cloud ↔ edge. |
…Phew. If you have read this far, you can already see through what is inside "the Tidy-Up Ghost." Now go watch the real tidying-up at the booth. *shiver… shiver…*
Reading / reference. We position the π0.5, GR00T, SmolVLA, and ACT covered in §2 of the main text by their relationships (lineage, openness, scale, use) to the major VLAs in the world. Numbers and public scope are as published as of June 2026 (sources at the end).
The term "VLA" itself was coined in July 2023 when Google DeepMind's RT-2 added "action" output to a VLM. In other words, all four main-text models stand on the arena that RT-2 defined.
The direct ancestor of the main-text models is ACT (main text, Stanford, 2023). The Action Chunking (predicting actions as a chunk) that ACT popularized is now a standard technique for all VLAs, and the direct bloodline runs its authors Levine and Finn founded Physical Intelligence the next year → π0 → the main text's π0.5 (see the "kinship" in §2.8). Furthermore, continuous action generation by flow matching / diffusion is "DNA" common not only to π0.5, GR00T, and SmolVLA but also to OpenVLA and RDT-1B described below.
2023.7 RT-2 (Google) ── named & established "VLA" ──┐
├─▶ the arena of the whole field
2023 ACT (main text / Stanford) ── invented Action Chunking
│ authors Levine/Finn founded
▼
2024-25 π0 → π0.5 (main text / Physical Intelligence)
+ shared flow matching ──▶ GR00T/SmolVLA (main text) / OpenVLA / RDT-1B
VLA has an even more deeply rooted "open culture" than LLMs. It splits broadly into three layers, and all four main-text models are on the "obtainable" side. The more cutting-edge the humanoid players, the more they hold their weights close.
Because this demo runs on a ¥40,000 arm and edge (Jetson), it leans small to medium. The frontier's giant models are at the right end of this figure.
small ◀─────────────────────────────────────────▶ large
Octo SmolVLA GR00T/π0.5 OpenVLA RT-2 PaLM-E
~93M ~450M(main) ~3B(main) 7B 55B 562B
└────── the 4 main-text models are around here ──────┘
(ACT is ~80M, a lightweight, no-language-conditioning method, even further left)
| Model | Developer | Openness | Scale | Relationship to the 4 main-text models |
|---|---|---|---|---|
| RT-2 | Google DeepMind | closed | ~55B | the source of the name "VLA." The conceptual parent of all 4 models |
| Gemini Robotics | Google DeepMind | not public | undisclosed | the closed rival of π0.5 and GR00T (full body too) |
| Helix | Figure AI | not public | 7B + 80M | the same two-layer System1/2 philosophy as GR00T / humanoid version |
| OpenVLA | Stanford et al. | fully open | 7B | the standard open model SmolVLA uses as its performance benchmark |
| Octo | UC Berkeley | fully open | ~93M | the small, open neighbor alongside SmolVLA |
| RDT-1B | Tsinghua University | open | 1.2B | the same diffusion family as π0.5 / bimanual-specialized |
| GO-1 | AgiBot (China) | partially public | undisclosed | the Chinese player in general-purpose foundations like GR00T and π0.5 |
| GR-2 / GR-3 | ByteDance | limited | undisclosed | confusingly named like GR00T, but a different thing |
| LBM | Toyota + Boston Dynamics | not public | undisclosed | on the full-body humanoid side, competing with GR00T |
| PaLM-E | closed | 562B | an ancestor from before VLA. A different lineage contemporaneous with ACT |
The four main-text models share the commonality of being "small to medium models within reach openly, connected to the lineage that descends from ACT, on the arena RT-2 created." The frontier's Gemini Robotics, Helix, and LBM are larger and step into full-body control, but are closed—which is precisely why this demo makes an enterprise-grade demo work with "4 obtainable models × a ¥40,000 arm." That is its position on the overall map.
Sources: RT-2 (blog.google) / Gemini Robotics (deepmind.google) / Helix (figure.ai) / OpenVLA (arXiv 2406.09246) / Octo (arXiv 2405.12213) / RDT-1B (arXiv 2410.07864) / GO-1 (arXiv 2503.06669) / GR-3 (arXiv 2507.15493) / LBM (Toyota pressroom) / PaLM-E (arXiv 2303.03378). Scale and public scope are as published as of June 2026.