Validation, Drift & Model Lifecycle
A learned component is not validated once and finished. Its authority is only as current as the last evidence that it still behaves — so the lifecycle, not the commissioning test, is the point.
This page is part of the AI & ML Integration section; read the authority gate first. Everything the register grants a method is conditional on the work below. The frameworks named here are process guidance, not functional-safety permission — none lets a learned component hold a safety function (see Safety & Security Boundaries). Review pending.
Why the lifecycle is the point
A learned model’s behaviour comes from training data, so it degrades as the plant, sensors, product, and environment drift away from what it learned. A model that was advisory-grade at commissioning can silently become worse than useless a year later, and nothing about its output announces the change. So the authority a method holds is perishable: it must be re-earned, or dropped when the evidence lapses. The lifecycle is the mechanism that keeps the authority claim honest.
The vocabulary: NIST AI Risk Management Framework
Two published NIST documents give this work a shared, non-proprietary vocabulary — used here for structure, not as evidence that a model may hold authority:
- NIST AI RMF 1.0 (NIST AI 100-1, Jan 2023) organises AI risk work around four functions — Govern, Map, Measure, Manage — and a set of trustworthiness characteristics. Voluntary guidance.
- NIST AI RMF: Generative AI Profile (NIST AI 600-1, 26 Jul 2024) is a “cross-sectoral profile of and companion resource for the AI RMF 1.0” (verified at NIST) — it does not replace it. It adds generative-AI-specific risks (for example confabulation — plausible, fluent, wrong output — plus information integrity, data privacy, and information security) and is the relevant companion whenever an LLM is in the architecture.
The lifecycle demands, mapped to the four functions
| Function | What it demands of a learned component |
|---|---|
| Govern | A named owner for the model and its authority; management of change on every retrain, update, or authority change; an audit trail that can reconstruct a decision from data, version, context, and human action. |
| Map | Data lineage (provenance, licence, collection conditions); the validated operating envelope and the known-invalid regimes; coupling and interface ownership inside a twin. |
| Measure | Leakage-free test sets split by held-out component, not window; uncertainty reported with every output; validation at the deployment conditions — hardware and numerical precision, not only the bench. |
| Manage | Drift / out-of-distribution monitoring against the validated envelope; a defined, hazard-analysis-derived failure response; retained, re-deployable model versions for fast rollback; human review on a cadence and on every anomaly. |
Two of these carry traps documented elsewhere in this section. Measure: split by held-out components — window-level or by-load splits inflate fault-diagnosis accuracy to near-100% while the honest number is near chance (see Model Families & Fit). Manage: a residual monitor is necessary but not sufficient — a PINN can relax to a plausible, wrong state while its residual stays low, so independent cross-checks matter.
The failure response is the envelope’s job
When the model is stale, low-confidence, out-of-distribution, unavailable, or contradicted, the non-learned envelope applies the hazard-analysis-derived response — never a universal default. This is the same envelope as the gate page:
flowchart TD
R["Model result
+ confidence · version · freshness"] --> C{"Fresh? in-distribution?
confident? corroborated?"}
C -->|Yes| USE["Use at the allowed authority level"]
C -->|No| FR["on_fail — hazard-analysis-derived"]
FR --> O1["Bumpless transfer to base controller"]
FR --> O2["Bounded-lifetime hold"]
FR --> O3["Manual takeover"]
FR --> O4["Defined safe state / shutdown"]
RB["Rollback to last known-good version"] -.-> R
classDef q fill:#e8eff8,stroke:#2b5797,color:#1e1e1e
classDef go fill:#eef4ed,stroke:#3a6b35,color:#1e1e1e
classDef warn fill:#fdf6e3,stroke:#8b6914,color:#1e1e1e
class C q
class USE go
class FR,O1,O2,O3,O4 warn
class RB warn
The model-evidence ledger
The durable artefact of all of the above is a per-component model-evidence ledger — one record that makes a learned component’s evidence auditable in one place: domain and phenomenon; governing assumptions and boundary conditions; parameters and their sources; datasets with licence and provenance; calibration and uncertainty; resolution; coupling and interface ownership; solver and deployment precision; validation experiments and acceptance limits; known-invalid regimes and fallback behaviour; and model version, authority level, owner, and change history.
→ Download the AI / model evidence ledger — a blank, fill-in template (also on the Engineering Templates page). Start it at design time and maintain it across the model’s life, rather than reconstructing it after an incident.
What this means for design
- Treat authority as perishable — schedule revalidation, monitor drift, and drop the authority when the evidence lapses.
- Split test sets by component, carry uncertainty, and validate at deployment conditions — bench accuracy is not field accuracy.
- Wire the failure response into the non-learned envelope, hazard-analysis-derived, and keep every model version rollback-ready.
- Keep a model-evidence ledger from day one, and put model changes through management of change.
- Use NIST AI RMF for structure and vocabulary (and the GenAI Profile for any LLM component) — remembering it is process guidance, not a route to safety authority.
Review status
Distilled from a corpus-owned Draft note and Review pending. NIST AI RMF is framed as process vocabulary, not safety permission. Return to the authority gate, Model Families & Fit, The Digital Twin, or Safety & Security Boundaries.
This site is an independent educational reference for industrial automation standards. It paraphrases and organizes guidance in original language; it is not a substitute for the official published standards, professional engineering judgment, or legal review, and content has not been independently verified against current published editions.
Items marked Review pending await technical review against the identified edition. Items marked Planned have no reviewed source material behind them. Do not rely on this site for compliance determinations, safety-critical design decisions, or legal interpretation.