Physical AI Foundations / 02

When Computer Vision Starts Understanding Events, Not Just Objects

Computer vision is moving beyond object detection toward streaming video, event understanding and distributed edge inference. The harder question is who defines what counts as an event and what the system is allowed to do next.

M.K.August 19, 202612 min read

A bright future waterfront city where roads, bridges and buildings integrate distributed sensing and edge computing into everyday infrastructure

M.K. / FIELD NOTESPhysical AI / Future Cities / 2026

Large language models gave AI a new way to read and manipulate the information humans have already produced. Computer vision is pushing in a different direction: it gives AI a way to keep watching a world that does not stop moving.

The important shift is not simply that cameras can recognize more objects.

It is that vision systems are beginning to move from answering, “What is in this frame?” toward answering, “What is happening here, and does it matter?”

Those are very different problems.

Detecting a person, a vehicle or a dog is perception. Deciding whether a vehicle is moving the wrong way, whether a package has been left unattended, whether two road users are entering a dangerous conflict, or whether an animal has crossed into a protected area requires time, relationships, context and rules.

Once a camera moves from recording objects to interpreting events, it becomes more than a recorder. It starts to function as a perception node inside a broader Physical AI system.

That is the transition I think matters most.

Computer vision is moving from labels to events

The classic computer vision stack is built around tasks such as image classification, object detection, segmentation, pose estimation and tracking.

These capabilities remain essential. A detector can identify three people, two cars and a traffic signal. A tracker can follow the same car across a sequence of frames.

But neither capability, by itself, tells a system what the scene means.

This is where newer video-language and multimodal systems are changing the architecture.

NVIDIA now describes Metropolis as an edge-to-cloud platform for turning video and sensor data from physical environments into actionable insights. Its current DeepStream stack supports real-time streaming analytics, multi-camera processing and generative-AI-assisted video workflows. Google’s current Gemini video-understanding documentation similarly shows models describing video, answering questions about content, extracting information and referring to specific timestamps.

The direction is clear: video systems are becoming less like collections of independent detectors and more like systems that try to reason over what is unfolding.

That changes the question from:

“Can the model identify a bicycle?”

to:

“Is this bicycle, the right-turning vehicle beside it and the pedestrian entering the intersection forming a dangerous interaction over the next few seconds?”

The second question is much closer to what real environments require.

The next camera feature is not resolution. It is an event model

Camera products have traditionally been compared through sensor size, resolution, lenses, frame rate, low-light performance and compression.

Those factors still matter. In fact, they remain hard limits on what an AI system can infer.

Arm’s 2026 work on low-light AI imaging is a useful reminder. Improving visibility, colour and detail in difficult lighting conditions matters because downstream vision systems can only reason about what the imaging pipeline preserves. A more capable model cannot recover information that the sensor and image-processing chain never captured reliably.

But the next layer of product differentiation is increasingly about the event model.

An event model defines what the system considers worth noticing.

A retail system may care about a queue that grows unexpectedly, a shelf that becomes empty or repeated movement through a restricted area.

A factory system may care about a person entering a hazardous zone, an object falling, or a process drifting outside a defined safety condition.

A traffic system may care about wrong-way movement, a near conflict, an obstruction or a sudden change in flow.

A trail camera may only need to keep clips containing certain animals instead of uploading hours of empty footage.

Arm’s current developer material includes a local AI smart-camera exercise using live camera input and object detection to drive application decisions. The important idea is not that every frame becomes “AI.” It is that the device can decide locally which observations deserve further processing.

I would define the value of the next generation of smart cameras this way:

They do not simply record more. They decide earlier what is worth understanding.

Edge AI turns vision into a distributed capability

If every camera uploads raw video continuously and every interpretation happens in a data centre, the architecture is conceptually simple.

In practice, it quickly runs into bandwidth, latency, storage, privacy and operating-cost constraints.

That is why computer vision and edge AI are likely to keep converging.

Arm positions its current edge AI stack around Cortex-M, Cortex-A and Ethos-U processors running inference on low-power and resource-constrained devices. NVIDIA’s Metropolis and DeepStream stack spans edge, on-premises and cloud deployments.

The more important point is architectural: not every frame needs to leave the device.

A practical vision pipeline may perform object detection, tracking, duplicate filtering or basic event checks locally, then decide whether to retain a clip, send metadata, or call a larger model for deeper reasoning.

That produces a stack that looks more like:

sensor → image signal processing → local perception → event filtering → selective upload → higher-level reasoning → action

This is one reason computer vision is such a useful entry point into Physical AI.

Vision is often the perception layer. Once perception can filter and interpret locally, the rest of the system no longer needs to move the full raw world into the cloud before it can make a decision.

The trade-off is that intelligence becomes distributed.

A small device may handle detection. A stronger local processor may handle tracking and event logic. A cloud model may handle long-horizon reasoning, cross-camera queries or tasks that need language and outside knowledge.

The competition is therefore not only about which model is best. It is about how the perception stack is divided.

Video is not a stack of images

One of the easiest mistakes to make in current AI discussion is to assume that video understanding is nearly solved because multimodal models can produce impressive descriptions of uploaded clips.

Continuous video is much harder than a static image.

Events have order. They have duration. They can be incomplete. The right answer at one timestamp may be “not enough information yet.”

That temporal problem shows up directly in research.

OVO-Bench, published at CVPR 2025, evaluates online video models under three conditions: tracing relevant past events, understanding what is happening at the current moment, and delaying a response until enough future information arrives. Its evaluation found a substantial gap between the tested Video-LLMs and human agents on online video understanding.

A separate CVPR 2025 study on temporal comprehension found that Video-LLMs could be sensitive to changes in video content, language queries and task settings, exposing consistency problems in temporal grounding.

Even current production documentation contains a useful reminder of the underlying constraint. Google’s Gemini video-understanding guide states that video is sampled at 1 frame per second by default and explicitly warns that fast action can lose detail.

That does not mean the technology is weak. It means “understanding video” is not one capability.

A person standing at a doorway for five seconds and a person standing there for forty minutes may represent completely different events.

A vehicle approaching a pedestrian while slowing down and the same vehicle accelerating toward the pedestrian are not the same risk.

A box sitting on the ground may have just been placed there or may have been abandoned for an hour.

A single frame can look almost identical in all three cases.

Real vision systems therefore need temporal intelligence, not only spatial intelligence.

Multi-camera vision becomes a problem of shared world coordinates

A single camera has one viewpoint. A network of cameras has a consistency problem.

The same person or object may appear at a different angle, scale, lighting condition and level of occlusion as it moves between cameras.

By August 2026, NVIDIA DeepStream 9.1 includes Multi-View 3D Tracking and AutoMagicCalib capabilities that map detections from multiple calibrated cameras into a shared coordinate system and maintain object identity across views.

The engineering significance is larger than “better tracking.”

The system is trying to maintain a persistent representation of physical space rather than treating every camera feed as an isolated stream.

Once multiple cameras in a road network, warehouse, factory or public space contribute to one shared state, vision starts to acquire something closer to spatial memory.

That can be extremely useful.

It also expands the governance problem.

The deepest risk is not only whether AI sees correctly

The obvious computer-vision risk is misclassification.

A model can miss a real hazard. It can label normal behaviour as suspicious. It can associate the wrong identity with an observation.

Those problems matter, and performance is not uniform across every environment or every population.

NIST’s Face Recognition Technology Evaluation is only one specific area of computer vision, so it should not be generalized to all vision systems. But its demographic evaluations remain an important example: false match and false non-match behaviour can vary by demographic group and image quality, and the evaluation continues to track those differences across submitted algorithms.

Still, I think there is a deeper design problem.

The question is not only:

“Did the model see correctly?”

It is also:

“What was the system told to notice?”

Imagine a camera system configured to detect “unusual loitering.”

Before model accuracy even enters the discussion, someone has to decide what counts as unusual. How long is too long? Which areas are restricted? Are employees treated differently from visitors? What happens after the event is triggered? How long is the record retained? Can the inference later be reused for another purpose?

Those are not purely computer-vision questions.

They are product, policy and governance decisions.

This is also why moving inference to the edge does not automatically solve privacy.

Keeping raw video on the device can reduce some transmission and centralized-storage risks. But a local device can still continuously classify people, infer behaviour and create identity relationships.

The governance question shifts from “Who stores the video?” to “Who is allowed to define and use the inference?”

A mature vision system therefore needs at least four things to be designed together:

perception accuracy, event definition, data minimization and downstream action boundaries.

A system can be technically impressive and still be unacceptable if any one of those layers is poorly designed.

The M.K. angle: the camera is becoming a local observer

If I pull the technology back to a system-level view, I do not think the most important story is that AI can finally “see.”

The more important transition is that the camera is changing roles.

A recorder captures evidence and leaves interpretation to people.

A local observer begins to decide, at the point of capture, what happened, whether it should be preserved, whether another system should be notified and whether more compute is justified.

That turns cameras and other optical sensors into distributed intelligence nodes.

I do not expect every camera to become a powerful general-purpose AI system.

The more likely architecture is layered.

Small edge devices handle detection and filtering. More capable local processors handle tracking and event interpretation. Cloud systems handle long time horizons, cross-camera reasoning, natural-language interaction and outside knowledge.

In other words, the future of computer vision may be determined as much by system partitioning as by model capability.

The full perception stack includes the sensor, ISP, local compute, model, memory, network, cloud reasoning and the policy that determines what actions are permitted.

Once AI can keep watching, the next question is unavoidable:

What should it be allowed to do?

Four signals worth watching next

The first is whether video-language models move from offline clip analysis to reliable streaming understanding.

The real test is not whether a model can summarize a finished video. It is whether it can maintain context while an event is still unfolding, revise its interpretation as new evidence arrives and recognize when it should wait before answering.

The second is whether edge hardware can support more capable visual reasoning within realistic power and thermal budgets.

If local systems can only perform basic detection while every meaningful interpretation still requires the cloud, architecture will remain constrained by network availability, latency and cost.

The third is whether evaluation expands beyond headline accuracy.

For real systems, temporal consistency, false-alarm rates, occlusion robustness, low-light performance, cross-camera identity consistency and error distributions across environments can matter as much as a single benchmark score.

The fourth is whether governance catches up with event understanding.

When a camera does more than save images and starts assigning meaning to behaviour, who defines the event, who can query it, who can review it and what action can follow become core product requirements.

Those questions may eventually matter more than resolution.

Frequently asked questions

Is computer vision the same thing as Vision AI?

The terms overlap, but they are not perfectly identical. Computer vision is the established research and engineering field covering tasks such as classification, detection, segmentation, tracking and scene understanding. “Vision AI” is often used more broadly in industry for applications that combine those techniques with deep learning, multimodal models, vision-language models and edge inference.

Will vision-language models replace object detectors?

Not across the board. Many real-time systems still need detectors and trackers that are fast, inexpensive and predictable. Vision-language models are more likely to be layered above them for natural-language queries, event summaries and contextual reasoning. The two approaches can coexist in the same system.

Why does computer vision need edge AI?

Video creates large amounts of data and many applications need low-latency responses. Edge inference can perform detection, filtering and some event logic near the sensor, reducing bandwidth use and cloud dependence. But edge processing does not automatically make a system private; inference policy and data use still require governance.

Do more cameras automatically make a vision system more accurate?

No. More viewpoints can reduce blind spots, but they also add calibration, synchronization, identity matching, occlusion, network and data-fusion problems. Without a consistent spatial and temporal model, an additional camera can simply add another conflicting stream.

What should buyers look for in an AI camera?

Beyond sensor quality, low-light performance and resolution, I would look at local inference capability, event definitions, false-alarm behaviour, temporal understanding, offline operation, data retention, auditability and the ability for a human to override or review consequential actions.