Final Proximity Space Systems

Let’s Talk Space

07Verification and validation in space systems

Building it right, and
building the right Thing.

The test pyramid from model in the loop to hardware in the loop, what each level can prove, and the gap that testing cannot reach at all.

T minus 00:08:00   TWO QUESTIONS

Verification is not validation

The distinction is old and worth keeping. Verification asks whether the system meets its specification. Validation asks whether the specification was right. A perfectly verified system built to a wrong requirement fails in flight, and the paperwork will show every test passing.

Rendezvous and docking is unusually exposed to the second failure, because the specification encodes assumptions about the target that nobody can check until arrival: how fast it tumbles, how reflective it is, whether it manoeuvres.

Mission control during the last day of Apollo 13. The spacecraft was verified against its specification; the situation it met was not in the specification, which is the difference between the two questions. NASA, S70-35368.

T minus 00:05:00   THE PYRAMID

Four levels, each proving something different

MIL

Model in the loop

The algorithm runs against a plant model, both in the modelling environment. Cheap, fast, and where the control design is actually settled: gains, horizon lengths, constraint margins and the shape of the approach corridor are all decided here, because this is the only level where a thousand variations cost nothing. Cannot show: anything about the implementation.

NASA SP-2016-6105, Systems Engineering Handbook

Rendered views of an approach in the Systems Engineering Simulator at Johnson. At this level the whole world is a model, which is what makes a thousand variations affordable. NASA/Bill Stafford.

SIL

Software in the loop

The real flight code, compiled, running against the same model on a workstation. Catches the gap between the model and the code: fixed-point behaviour, integer overflow, initialisation order. Cannot show: timing on the target processor.

NASA NPR 7150.2D

The engine software verification laboratory at Marshall in 1999, where the flight code controlling the Shuttle main engines ran against a simulated engine before every flight. NASA/MSFC.

PIL

Processor in the loop

The flight code on the actual flight processor, plant still simulated. This is where worst-case execution time becomes a measurement rather than an estimate. Cannot show: sensor and actuator behaviour.

ECSS-E-ST-10-02C, Verification

The Shuttle Avionics Integration Laboratory at Johnson, a skeletal orbiter of real flight avionics fed simulated inputs, photographed in July 2011 during the final Shuttle mission. NASA/Smiley N. Pool.

HIL

Hardware in the loop

Real sensors, real actuators, real timing, stimulated by a simulated world. The last level before flight, and the most expensive. Cannot show: the things only the flight environment produces, meaning real lighting, real thermal cycling and real radiation.

NASA, Stennis Space Center

The SLS core stage Green Run hot fire, January 2021. At the top of the pyramid the test article is the vehicle itself. NASA/Danny Nowlin.

T minus 00:02:00   BEYOND TESTING

What testing cannot reach

Every level above samples the input space. For a controller that must hold constraints across a continuum of states and disturbances, sampling cannot establish coverage. Passing ten thousand Monte Carlo runs says nothing rigorous about the run you did not do.

Two things close that gap. Reachability analysis computes the set of states the system can reach under bounded disturbance, and asks whether that entire set stays clear of the keep-out zone. Robust control formulations such as tube MPC carry the uncertainty inside the optimisation, so feasibility is a property of the controller rather than an observation about the tests.

Monte Carlo tells you the system worked in the cases you sampled. A reachable set tells you it works in every case within your assumptions, and forces you to write down what those assumptions were, which is often the more valuable output.

A manual deployment test of the Webb Pathfinder test article, not the flight telescope, outside Chamber A. Some verification can only be done by touching the hardware. NASA/Desiree Stover.

T minus 00:00:20   SOURCES

Where this comes from

ECSS-E-ST-10-02C Rev.1, Space engineering: Verification. ECSS Secretariat, ESA-ESTEC, 1 February 2018.

NASA Systems Engineering Handbook, NASA SP-2016-6105 Rev 2.

NPR 7150.2D, NASA Software Engineering Requirements. NASA, effective 8 March 2022.

Mayne, D. Q., J. B. Rawlings, C. V. Rao and P. O. M. Scokaert. “Constrained Model Predictive Control: Stability and Optimality.” Automatica 36, no. 6 (2000): 789–814.

Mayne, D. Q., M. M. Seron and S. V. Raković. “Robust Model Predictive Control of Constrained Linear Systems with Bounded Disturbances.” Automatica 41, no. 2 (2005): 219–224.

Althoff, Matthias, Goran Frehse and Antoine Girard. “Set Propagation Techniques for Reachability Analysis.” Annual Review of Control, Robotics, and Autonomous Systems 4 (2021): 369–395.

Rawlings, James B., David Q. Mayne and Moritz M. Diehl. Model Predictive Control: Theory, Computation, and Design. 2nd ed. Madison, Wisconsin: Nob Hill Publishing, 2017. Book site.

All nine topics
PreviousECSS and NASA guidelines for space software NextGround tests for proximity operations