The Tidy-Up Ghost

People often say "the era of Physical AI has arrived." For Amazon, that is not a prophecy about the future but an everyday reality.

We operate more than one million robots in warehouses around the world, and we run a foundation model (DeepFleet) that coordinates the entire fleet — all at the scale of our own operations. We are a company that confronts the difficulty of "running the physical world with AI" every day, and solves it every day. What we built through that implementation and operation is now open in the form of AWS.

Compressing months of training into a few hours with GPU simulation, transferring the trained brain to real hardware (Sim2Real), running it on the edge with low latency, and operating many units together — you can assemble the entire long journey of Physical AI, end to end, on a single foundation. That is what it means to choose AWS.

When a visitor speaks into the microphone, the character "the Tidy-Up Ghost" responds, and the SO-101 robot arm inside a miniature house automatically puts away erasers and markers — an Edge-to-Cloud Physical AI demo.

• A configuration that loosely couples cloud intelligence (conversational agent and VLM-based success/failure judgment) with edge real-time control (VLA inference on Jetson Thor) via AWS IoT Core (MQTT).
• The robot's motion model is a VLA (Vision-Language-Action) model fine-tuned with imitation learning, built as an end-to-end robot-grade MLOps pipeline: teleoperated data collection → training on SageMaker → distribution to the edge via Systems Manager + S3.
• Composed of the educational robot SO-101 (~¥40,000 (approx. US$270) street price) plus the inference edge computer NVIDIA Jetson AGX Thor developer kit (~US$3,499 (~¥500,000-range)). The arm itself is inexpensive, and the aim is to show that an enterprise-grade demo can be assembled using only AWS managed services.

An experiential demo that solves the everyday problem of "tidying a messy house is hard — wouldn't it be nice if a robot did it for you" with Physical AI and an IoT-enabled robot.

• An SO-101 arm is installed inside a miniature house (aluminum-frame enclosure), and the robot puts items away with nothing but the visitor's voice commands.
• The robot does not run pre-programmed fixed motions; it moves with a VLA model that generates actions from camera images and language instructions (= demonstrating that "it moves because it was trained").
• After acting, the robot itself evaluates the overhead camera image with a VLM, judges success/failure, and speaks (an autonomous closed loop).

It consists of four layers: cloud (real-time control), edge (Jetson Thor), training pipeline (offline), and the physical booth.

PointsKey design points

• The cloud side is fully serverless/managed (Strands Agents, Bedrock, Polly, IoT Core, CloudFront, S3).
• Cloud and edge are loosely coupled via IoT Core MQTT. The cloud agent publishes natural-language commands, and the edge Strands Robots subscribes continuously and triggers VLA inference.
• VLA inference runs locally on the edge (Jetson Thor), so the arm can be controlled in real time without being affected by network latency.
• PLANNED A concept in which success/failure judgment is done by the cloud VLM (Bedrock) using the overhead camera image. Intelligence in the cloud, instant response at the edge. *Not implemented in the current code.
• CONCEPT ONLY Cloud ON/OFF control of the lighting was an early request, but the current code does not implement it. The LEDs are simply always on via USB power.

2.1Implemented E2E architecture (code-verified)

The diagram above is an integrated view including plans. The diagram below is the implemented E2E architecture, confirmed by close reading of the repository as the configuration actually implemented in code. Inference, conversation, and IoT control are all completed within the single ap-northeast-1 region (only the training system is in a separate region, see §06). The conversational agent runs as a Bedrock AgentCore Runtime (Strands Agents + FastAPI WebSocket running on an ECR container), calling Transcribe, Bedrock Claude, and SageMaker TTS, and publishing commands to the edge via IoT Core.

Reference implementations / blogs

• AWS Blog "Building intelligent Physical AI from edge to cloud with Strands Agents, Bedrock AgentCore, Claude 4.5, NVIDIA GR00T, and Hugging Face LeRobot"
• Strands Labs blog (experimental agent development)

The full sequence from a visitor saying "put the eraser away" to the robot moving and speaking the outcome.

PointsCloud only at the round-trip junctions

• The round trip: voice command → intent interpretation in the cloud → command to the edge via MQTT → VLA inference and arm control at the edge.
• The VLA inference loop itself completes entirely within the edge (repeatedly acquiring camera video → outputting joint angles), so communication with the cloud occurs only at the "command" and "completion notice" junctions.
• The VLM success/failure judgment after the action (bottom of the diagram) is planned and not implemented in the current code (see the §02 implemented diagram). What is implemented is "voice → Claude conversation → MQTT → edge"; the judgment loop is not yet connected.

Through the app screen, visitors move back and forth between the following modes while conversing with the "Tidy-Up Ghost." Conversation choices can be selected by speaking into the mic, with mouse operation available as a fallback.

5.1The three modes

5.2The three tasks the VLA performs

* Before each task, an exhibit staff member manually sets up the scene.

1. Put the eraser in the basket (the most mature task; executable with just one arm, so it is the fallback for degraded operation)
2. Stash the marker in the "S3 bucket" pen holder (a little pun on Amazon S3)
3. Roll the dice toward a bowl, alternating left and right arms: before rolling, the visitor predicts even/odd, and after rolling, the VLM reads the pips and reports the result (a flourish to show off the VLM's recognition)

5.3Character production "the Tidy-Up Ghost"

• Speech synthesis is a voice clone via Qwen3-TTS (SageMaker inference EP), giving it a distinctive character voice (with the verbal tic "~anyo" at the end of phrases, a *shiver… shiver…* kind of vibe).
• Variations of success voices and failure voices are prepared to heighten the demo's experiential and entertainment value.

The heart of the demo is that "the robot moves because it was trained." It cycles through imitation-learning data collection by teleoperation → FT in the cloud → delivery to the edge. Sim2Real is also planned for integration.

Note: the learning method is imitation learning, not reinforcement learning. The Sim (Isaac Sim) is not a place for reward-driven trial and error; it is used for automatic generation of demonstration data, parallel collection, and pre-validation.

Implemented training pipeline (code-verified)

The diagram below is the implemented version, confirmed by close reading of the shimatte-ai-training repository. The training/research system all runs in us-west-2 (research account). SageMaker Training trains π0.5 / ACT / GR00T (N1.5–N1.7 family) / SmolVLA (ml.g5.2xlarge, with a custom ECR image only for GR00T), managed with SageMaker MLflow and Model Registry. Edge delivery of trained models via IoT Greengrass is a future configuration (not implemented at the time of this demo). ACT is adopted for the live show on real hardware (π0.5, GR00T, and SmolVLA are also validated on real hardware). Reinforcement learning (Isaac Sim + AWS Batch) is a Phase 5 plan (dashed).

Sense of data scale

• To make training robust, lighting patterns (partial on, all on, all off) are varied to give the data variation.
• The entire training process is visualized on the training dashboard (delivered via CloudFront, described later), and on the day this screen is used to explain "how the VLA is trained."
• Sim2Real: A Sim-based data collection environment is built in Isaac Sim (changing the standard SO Arm's wrist camera, mount, and gripper to the real-hardware spec). The aim is to supplement the labor-intensive nature of real-hardware data collection.

7.1Booth physical layout

7.2The on-site 3-screen setup

• The total is on the order of ~¥30,000–40,000 (approx. US$200–270) (main body, 3D-printed parts, aluminum frame, cameras, lighting, etc., excluding the Jetson Thor).
• For power, four SO-101 AC adapters in parallel fit perfectly into the frame.