Physical AI Foundations / 06

Why Robotics Is Hard: The Real World Has No API Contract

Robotics is difficult not simply because models need to improve, but because perception errors, contact physics, latency, mechanical wear and human behaviour accumulate into uncertainty that the system must manage safely.

M.K.September 6, 202612 min read

A quadruped robot moving through rocky terrain and vegetation at the edge of a near-future waterfront city, with autonomous transport and drones operating in the distance

M.K. / FIELD NOTESPhysical AI / AI Infrastructure / 2026

Software engineers work in a surprisingly luxurious environment.

APIs have documentation. Inputs have types. Services return status codes. Databases can be retried. Test environments can be rebuilt. Even when an external dependency fails, we can usually time out, roll back, retry, or queue the request for later.

Robots do not get any of those guarantees.

A floor does not expose its coefficient of friction. A cardboard box does not return an error saying its grasp point has deformed. Transparent glass does not promise to appear correctly in every depth sensor. A loose fastener does not emit a webhook. And a person walking into a robot's path does not announce the event through an API.

This is one of the things I think robotics discussions underestimate most often.

The hard part is not only getting an AI system to know what it should do. The harder engineering problem is building a system that can keep deciding whether it is still safe to act in a world with no API contract.

That follows directly from the broader argument in my article on Physical AI. Physical AI connects perception, reasoning, planning and action. Robotics makes the consequences of that connection concrete because uncertainty at one layer propagates into the next.

Software has contracts. Physics does not.

A software contract usually tells us something about valid inputs, outputs, failure conditions, timeouts, versions and responsibility boundaries.

The physical world offers almost none of those guarantees.

Consider a robot arm reaching for a cup. In a demo, the sequence looks simple: detect, approach, grasp.

In the real system, the estimated cup position may be slightly wrong. The surface may be reflective. The handle may be occluded. The table height may differ from calibration. Gripper material may have worn enough to change friction. Liquid in the cup changes the centre of mass. Contact can push the object away from the pose the planner expected.

Those are not exceptional edge cases.

They are normal properties of physical environments.

Google DeepMind's Gemini Robotics-ER 1.6, released in April 2026, emphasized spatial reasoning, multi-view understanding and success detection for exactly this kind of embodied problem. That model has since been superseded in Google's current developer stack by Gemini Robotics ER 2, so I would not treat 1.6 as the current production reference. The more important point is the direction of travel: the current Gemini Robotics 2 family places increasing emphasis on long-horizon task progress, self-correction, uncertainty resolution and safety around humans.

That suggests a useful distinction. Better robotics AI is not only about doing more things. It also needs to become better at recognizing when conditions no longer support the intended action.

Perception is not only about what the robot sees

Computer vision often discusses performance in terms such as accuracy, precision and recall.

Robotics adds a more severe consequence:

A perception error can become a motor command.

If a vision system estimates an object's position incorrectly, a classification system may simply produce a worse metric. A moving robot can turn the same estimation error into a collision, a missed grasp or a damaged object.

Occlusion makes the problem harder. A robot's own arm can block its camera. After it moves an object, the geometry of the scene changes. Multiple cameras can reduce blind spots, while force and tactile sensing can compensate for some visual uncertainty. But each added sensor also creates calibration, synchronization and sensor-fusion problems.

So I think the most useful perception output in robotics is not merely:

What is this object?

It is closer to:

What do I think is happening, how certain am I, what can I not observe, and is the remaining uncertainty low enough to continue acting?

This is why embodied reasoning matters. A useful robot cannot stop at scene description. It needs to reason about spatial relationships, viewpoint, object state and the likely consequences of its own actions.

The moment contact begins, the problem becomes physics

Some of the most impressive robotics demos happen before contact: a robot identifies an object, plans a trajectory and moves smoothly toward it.

But many of the hardest problems begin after contact.

Inserting a plug, turning a key, folding fabric, rotating a handle, fitting a tight mechanical part, manipulating a cable or moving through loose terrain are not solved by reaching the right coordinate. Friction, deformation, reaction forces and geometric error create new states as soon as the robot touches the world.

NVIDIA's current Newton work explicitly targets contact-rich manipulation and locomotion problems including tight-tolerance assembly, in-hand manipulation, cloth, cables and granular terrain. Isaac Sim and Isaac Lab similarly treat physics simulation and robot learning as important tools for reducing the gap between training environments and real deployment.

The deeper point is straightforward:

A robot cannot only predict what the world will do. It has to keep measuring what its last action actually caused.

That is why mature robotics systems need closed-loop behaviour.

Not plan once and execute once, but something closer to:

observe → act a little → observe again → update state → correct → stop or recover when necessary

The quality of that loop matters more than how intelligent one isolated action looks in a demonstration.

Latency is a control problem, not a UX problem

In a web application, a few hundred milliseconds of delay is usually a user-experience issue.

In robotics, delay changes the state being controlled.

Sensors have latency. Image pipelines have latency. Inference has latency. Networks have latency. Controllers run at their own cycle times. Worse, those delays are not always constant.

If a robot acts on an old perception of the scene, it is effectively controlling a world state that may no longer exist.

This is one reason Edge AI becomes structurally important in Physical AI. It does not mean every model should run locally. It means safety-critical loops need a clear boundary around which decisions cannot depend on an unpredictable remote round trip.

Local inference is not a universal answer either. Smaller models may lose capability. Accelerated inference still consumes power and thermal headroom. Sensors remain noisy.

The architecture therefore has to separate time scales.

Fast safety and low-level control often belong in deterministic mechanisms with predictable timing. Slower perception, planning and reasoning can operate at a different cadence. Cloud systems can handle heavier analysis, model updates and tasks that do not require immediate reaction.

This is not one model solving everything. It is systems engineering.

A robot's body changes underneath the software

Software systems encourage us to think that if the version has not changed, behaviour should remain roughly stable.

Robots are different.

Tyres wear. Joint backlash changes. Gripper surfaces age. Camera lenses get dirty. Batteries degrade. Motors heat up. Mechanical structures loosen. The same command can produce a different physical result over time even when the software is unchanged.

This is one of the most important differences between Physical AI and purely digital AI:

The model is not controlling a fixed execution environment. It is controlling a body that ages.

That changes what observability should mean. Robot observability cannot stop at software logs. It may also need actuator current, temperature, cycle count, calibration drift, collision history, sensor health and long-term changes in task success.

In my previous article on AI-device product lifecycle management, I described hardware, runtimes, models, OTA and support periods as a dependency chain that evolves after shipment. Robotics adds another moving dependency: the physical state of the machine itself.

Safety cannot depend on the model being right most of the time

When an AI system only generates text, a human can often see the error before anything physical happens.

When the output controls motors, safety architecture has to assume the model can be wrong.

ISO 10218-1:2025 and ISO 10218-2:2025 make that systems perspective explicit for industrial robots and industrial robot applications. The standards address safe design, risk reduction, integration, commissioning, operation and maintenance rather than treating model accuracy as a substitute for system safety.

The scope matters. ISO 10218 is focused on industrial robotics. The current ISO pages explicitly exclude service robots used by the public, consumer products, healthcare robots and several other categories. It should not be generalized as if it automatically governs every future home or public-space robot.

That limitation is itself instructive.

Robots leaving controlled industrial environments for homes, shops, streets and care settings face a harder safety problem because engineering teams no longer control the surrounding environment.

For that reason, I think an increasingly important robotics capability is graceful uncertainty handling.

A system should be able to recognize that it cannot see clearly, that a grasp did not match the expected result, that a task appears infeasible, that conditions may be outside a safe operating region, or that it needs to slow down, retry, switch strategy or hand control back to a person.

DeepMind's July 2026 Gemini Robotics 2 release makes this direction unusually concrete. Its ASIMOV-Agentic evaluation includes whether the embodied reasoning agent can predict task feasibility, reject unsafe tool calls and proactively request human intervention when uncertain. DeepMind also describes human-proximity handling that can trigger safety tools and bring a robot to a safe stop.

The point is not that these capabilities solve robotics safety. They show where the engineering target is moving: uncertainty itself has to become part of the control problem.

M.K. Angle: robotics accumulates uncertainty debt

Software engineering has technical debt.

I think robotics has another form of debt that is easier to hide. I call it uncertainty debt.

This is my analytical framework, not an established industry term.

Uncertainty debt accumulates when a system can operate, but key physical uncertainties have not been measured, bounded or converted into explicit engineering decisions.

How much does vision degrade in low light? At what level of gripper wear does reliability begin to fall? How much network delay is allowed before a task switches to a local mode? How slippery can the floor become before speed must be reduced? How long can a sensor be occluded before the last known state is no longer trusted? After repeated grasp failures, should the system retry, change strategy or stop?

A demo can still look impressive even when none of these questions has a clear answer.

But the system is borrowing against future reliability.

The robotics companies I would expect to scale most reliably will not only compete on how many tasks their models can perform. They will compete on how effectively they convert uncertainty into measurable operating envelopes, failure modes, recovery policies and safety boundaries.

In that sense, the moat in Physical AI may not be simply having the smartest model.

It may be knowing when the model is not reliable and still having a system that can recover safely.

Simulation helps find failures. It cannot certify reality.

Simulation matters because real-world testing is expensive, slow and sometimes unsafe. It can expose policies to rare situations, vary physical conditions and reveal obvious failure modes before hardware is put at risk.

NVIDIA Isaac Sim and Isaac Lab emphasize physics simulation, synthetic data, domain randomization and robot learning. Research efforts such as ManipulationNet also highlight a persistent robotics problem: real-world manipulation is variable, while scientific benchmarking needs experiments that can be reproduced and compared.

But simulation should not be treated as proof that the physical world will behave the same way.

Its strongest role is to search for failure modes at scale, expand the range of conditions a system experiences and remove obvious weaknesses before field testing.

The final loop still has to return to real hardware.

No simulator can guarantee that every deployed robot will avoid tyre wear, sensor contamination, wet floors, loose fixtures, changing payloads or unexpected human behaviour.

What matters next is not which robot demo looks most human

I would watch four signals over the next few years.

First, do robotics benchmarks move beyond single-task success toward long-horizon reliability, recovery rate, failure diagnosis and uncertainty calibration?

Second, do foundation models settle into a clearer division of labour with deterministic safety and control layers, rather than trying to span every time scale inside one model?

Third, can simulation, fleet telemetry and field failures form a tighter feedback loop so that newly observed failures quickly become new test and training conditions?

Fourth, as service robots enter public and domestic environments, do safety standards, liability boundaries, support obligations and procurement requirements begin to matter as much as model benchmarks?

If those shifts continue, robotics competition will gradually move from a simple question:

Can the robot do the task?

Toward a harder one:

Can it keep doing the task safely when the conditions are not what the engineers expected?

That is the closest thing the physical world gives us to an API contract.

The world does not promise to follow our specification. The robot system has to promise that when the world does not behave as expected, it still knows what to do next.

Frequently asked questions

Why is robotics harder than ordinary AI software?

Because model outputs become physical actions, while sensing, friction, contact, wear, latency and human behaviour keep changing the system state. A robot therefore needs not only good predictions, but safe recovery when predictions or assumptions are wrong.

Can simulation eliminate the sim-to-real gap?

No. High-fidelity physics, domain randomization and synthetic data can reduce the gap and reveal more failure modes, but deployed hardware still experiences material variation, wear, sensor noise and human behaviour that must be validated in the field.

Can larger VLA or foundation models solve robotics by themselves?

They can improve scene understanding, planning, adaptation and task generalization, but they do not remove the need for control loops, timing guarantees, mechanical design, safety mechanisms, sensor health monitoring and failure recovery.

Is an emergency stop enough for robot safety?

No. Emergency stop is important, but a mature safety architecture also has to address risk reduction, integration, degraded modes, sensor failures, human proximity, uncertainty handling and recovery policy. Current industrial robot standards treat safety as a lifecycle and system-level problem.

What does uncertainty debt mean in this article?

It is an M.K. analytical framework, not an industry standard. It describes uncertainty that has not yet been measured or converted into an explicit operating limit, failure condition or recovery rule. That debt may stay hidden in demos, but it becomes a reliability and scaling problem in deployment.