Spatial AI Is Not a Model: How Engineering Teams Build Autonomous Systems
Mohsen is a seasoned optimization expert who takes a principal role in implementing and validating data science and artificial intelligence models.
PhD in AI • Senior Consultant #MachineLearning #DataScience @BasicFitNL Previous: @Shell @LINKITGroup @FU_Mashhad @TUDelft @ErasmusMC @DanielDenHoed @CapgeminiNL @KLM
A tour of the modern autonomy stack, one trade-off at a time: sensing, state estimation, planning under uncertainty, and the constraint budget that shapes all of it.
Spatial AI is having a moment. Fei-Fei Li calls spatial intelligence AI's next frontier, and her company World Labs is building world models that generate and reason over 3D scenes. Niantic Spatial is training a Large Geospatial Model on millions of scanned locations. Most of the coverage shares an assumption: spatial AI is a model, and once the model is good enough, machines will understand space.
Teams that ship autonomous systems (drones, rovers, warehouse robots, inspection platforms) learn a different lesson. Spatial AI is not a model. It is a pipeline of engineering decisions, made under hard constraints, where each choice reshapes what every later stage can do. A model is one component in that pipeline, and often the least troublesome one.
This article walks through that pipeline as a framework I will call the modern autonomy stack. At each layer, the useful question is the same: what trade-off is the engineering team making, and what does each side cost?
The modern autonomy stack
Nearly every autonomous system, whatever it looks like from the outside, is organized around four layers.
Sense. Sensors produce measurements: pixels, range returns, inertial rates.
Understand. Estimation turns measurements into a usable picture: where am I, what is around me, what is it made of.
Decide. Planning chooses an action given that picture and its uncertainty.
Act. Control turns the chosen action into motor commands, and the loop closes: acting changes what the sensors see next.
The idea has deep roots. Andrew Davison's 2018 FutureMapping paper defined Spatial AI as the perception half of this loop: simultaneous localization, dense mapping, and semantic understanding, in service of action. More recently, researchers published a blueprint for a unified autonomy stack organized around the same sense-think-act partition, spanning aerial and ground robots.
The partition survives because it is useful. Each layer produces an estimate the next layer consumes, so the interfaces are where teams isolate failures, swap components, and assign ownership. Alongside the four layers runs a fifth element that most architecture diagrams leave out, even though teams argue about it more than about any layer: a shared constraint budget of latency, power, thermal headroom, weight, and safety margin. No layer owns it. Every design choice draws on it.
Sense: choosing what the system is allowed to know
Everything downstream is bounded by what the sensors can measure, so the trade-offs start here.
Camera vs LiDAR is the canonical one. Cameras are cheap, light, low-power, and carry dense semantic information, but depth must be inferred and performance depends on lighting. LiDAR measures geometry directly and works in the dark, but costs more in money, weight, and power, and returns geometry with almost no semantics. Radar keeps working in conditions that degrade both, at coarse resolution. An IMU delivers motion at hundreds of hertz and drifts within seconds on its own. The resolution in most fielded systems is a suite chosen so that failure modes do not correlate: the conditions that blind one sensor should leave another usable.
Early vs late fusion. With multiple sensors, the team must decide where to combine them, and each fusion level buys something different:
| Fusion level | What you keep | What you pay |
|---|---|---|
| Early (raw measurements) | All cross-sensor information | Tight calibration, high compute, coupled failures |
| Feature level | Cross-sensor structure at manageable rates | Hand-designed interfaces between pipelines |
| Late (per-sensor outputs) | Modularity, per-sensor validation | Information discarded before it can help |
Benchmark accuracy rarely settles this choice. Calibration burden and failure isolation usually do: early fusion extracts the most from the hardware, and it also means one miscalibrated sensor can corrupt the whole estimate.
Understand: from measurements to a world
Two decisions dominate this layer: how to estimate state, and how to represent the world.
Classical estimation vs learned perception. The Kalman filter family and factor-graph methods carry explicit uncertainty, run on trivial compute, and rest on decades of theory. They require hand-specified dynamics and observation models, and they struggle with high-dimensional raw data. Learned perception consumes raw pixels and produces semantics no filter can, but its confidence estimates are unreliable and its failure modes are hard to characterize. The field is converging on hybrids: differentiable filters that keep the probabilistic structure and learn the models inside it, and recursive filters designed for out-of-distribution inputs. In a shipped system the two coexist: a classical estimator keeps the vehicle localized and upright, while learned models interpret the scene.
Sparse vs dense, occupancy vs semantic. The SLAM literature describes a capability ladder: sparse localization, then dense mapping, then semantic understanding. A sparse map is enough to know where you are and costs almost nothing to store; it gives no basis for interacting with anything. A dense map supports inspection and manipulation and consumes memory and compute continuously. An occupancy grid answers "can I move here" cheaply; a semantic map answers "what is this and does it matter" at the price of running inference and keeping the labels current. Experienced teams choose the poorest representation the task tolerates, because every step up the ladder is paid out of the constraint budget for the life of the product.
Decide: planning when the world is a probability distribution
The understand layer outputs estimates with uncertainty attached. The central decision in planning is what to do with that uncertainty.
The cheap approach thresholds it: treat a detection above some confidence as true, below it as absent, and plan in the resulting deterministic world. This works until an overconfident wrong estimate meets an action that cannot be undone. The alternative propagates uncertainty into the planner, which can then keep wider margins where the state is poorly known, and can choose actions whose purpose is to reduce uncertainty (move closer, look again) when the expected information justifies the energy.
MPC vs learned policy. Where a good dynamics model exists, model predictive control gives constraint handling and inspectable failures: when it misbehaves, you examine the model, the cost function, or the horizon. A learned policy can handle dynamics too messy to model and can be far cheaper per decision, but its behavior outside the training distribution is unknown, which matters exactly when things go wrong. A second trade-off sits inside the first: a longer planning horizon improves decisions and costs compute on every cycle, and on embedded hardware the planner shares that budget with perception. In domains with well-characterized physics, hybrids constrain the learned parts to physically plausible behavior; greenhouse climate control is one worked example of physics-aware models doing exactly this.
Act: paying for the loop with latency and power
A decision matters only at the rate the system can act on it, and closing the loop is where the constraint budget stops being abstract.
Edge vs offboard inference. Offloading compute to a server buys arbitrary model capacity at the price of link latency and link reliability. Keeping it onboard means living inside tens of watts of compute on an aerial platform, plus the thermal and weight cost of dissipating that power. The split most teams land on: anything inside the control loop (stabilization, obstacle avoidance) runs onboard, and anything that tolerates seconds of delay (map refinement, fleet analytics) can leave the vehicle.
Making models fit onboard is its own discipline. INT8 quantization commonly shrinks a model about four times and speeds inference two to four times, and benchmark studies of compact detectors on embedded modules report real-time frame rates only after that kind of optimization. The upstream consequence is the point: the perception team's architecture choice was never free, because every parameter is eventually paid for in watts and milliseconds at this layer.
Failure modes: where these systems break in the field
Demos end where fielded systems begin. Most of the engineering effort in a production spatial AI system goes into five problems that rarely appear in write-ups.
Calibration drift. Fusion assumes the sensors agree about where they are relative to each other. Vibration, thermal cycles, and small impacts change those extrinsics over weeks of operation, and the fusion degrades without throwing any error; the estimates just get quietly worse. Mature systems monitor calibration residuals and re-estimate extrinsics online instead of trusting the factory values.
Degraded sensors. Dust, condensation, glare, and fog produce data that is present and wrong. A sensor that knows it is blinded is a manageable input; a sensor reporting confident garbage poisons the estimator. Detecting degradation is a perception problem of its own, and it has to run inside the same compute budget as everything else.
Uncertainty mis-propagation. Overconfident perception is more dangerous than inaccurate perception. A planner told "the obstacle is exactly there" plans tighter maneuvers than one told "the obstacle is somewhere around there", and when the confident estimate is wrong, the failure is silent: every module performed to spec, and the defect lives in the interface contract between them.
Out-of-distribution scenes. Learned components fail confidently on inputs unlike their training data, and the field supplies such inputs endlessly: new lighting, new geometry, new objects. The practical mitigations are runtime monitors that flag unfamiliar inputs and fallback behaviors (slow down, hold position, return) that buy time; filter architectures designed for out-of-distribution operation attack the same problem at the estimator.
Validation. Teams that ship run a staged sequence: scenario-based simulation, then controlled field trials, then shadow deployment where the system decides but does not act, with regression tests against logged data every time a model changes. Coverage is counted in scenarios rather than operating hours, because accumulated mileage mostly resamples the easy cases.
Where the stack runs today
The same four layers run everywhere autonomy ships; what differs by domain is the constraint budget, and therefore which side of each trade-off the team is forced onto. Autonomous driving has large power budgets and extreme safety requirements. Warehouse robots work in a structured world under tight unit cost. Infrastructure inspection gives up GPS and works from vision and inertial sensing alone.
Controlled-environment agriculture compresses the whole argument into one machine. A drone flying inside a greenhouse gets no GPS, flies through reflective glass structures among plants that change shape daily, operates in humidity and dust that degrade sensors, and must run its perception onboard because the control loop cannot wait on a wireless link. A concrete account of running perception under those constraints is in Edge AI for greenhouse automation from Growmatics, a Dutch company building this stack for autonomous growing.
