AI Continuous Assurance Monitor
Enter an AI system’s current monitoring readings — hallucination rate, citation and retrieval accuracy, evaluation pass rate, tool-call success, drift, human override rate, policy violations — and get a deterministic assurance state (assured / monitoring-alert / reassessment-required / suspended) and the action it triggers. A safety-, integrity-, or policy-critical breach suspends the system; any other breach requires reassessment. A scoping aid, not a validated assurance system.
OUTPUT
TIME
Limitations — read before you rely on this
- This is a scoping aid, not a validated system, and not a monitoring platform. Reproduce the thresholds and state logic in a monitoring system you control; a point-in-time verdict is not continuous monitoring.
- It evaluates a single snapshot. Real continuous assurance watches trends over time — a signal drifting toward its threshold matters before it crosses, which a one-shot reading does not capture.
- The thresholds are SPEQ defaults, not regulatory limits. Set yours from the baseline your system was qualified against; a threshold copied unexamined can read healthy while the system degrades, or cry wolf on a well-behaved one.
- It only knows the signals you enter. A failure mode you do not monitor is invisible here — the state is only as complete as the monitoring behind it.
WHAT THIS CALCULATES
The current assurance state of a deployed AI system from its monitoring signals, and the action that state requires. It reads each signal — hallucination rate, citation and retrieval accuracy, evaluation pass rate, tool-call success, drift, human override rate, policy violations — against its thresholds and returns a deterministic state (assured / monitoring-alert / reassessment-required / suspended). It turns "the dashboard looks fine" into a decision: keep monitoring, investigate, reassess, or suspend.
THE METHOD
state = suspended if any critical_signal breaches; else reassessment-required if any signal breaches; else monitoring-alert if any signal warns; else assured- state
- the assurance state from the reserved vocabulary — not-assessed, assured, monitoring-alert, reassessment-required, or suspended
- critical_signal
- a signal whose breach is a safety, integrity, or policy failure (hallucination, citation accuracy, evaluation pass rate, policy violations) — flagged suspend-on-breach
- breaches
- a reading past its critical threshold — for a rate, above it; for an accuracy, below it
- warns
- a reading past its warning threshold but not yet its critical threshold
Each signal has a direction (higher-is-worse for rates, lower-is-worse for accuracies) and default warn/critical thresholds that are a SPEQ starting operating point, not regulatory limits — set yours from your qualified baseline. Monotonic: a worse reading never improves the state. Unknown signals are ignored.
THE INPUTS, AND WHAT THEY MEAN
- Signal readings
- The current value of each monitored signal over your window. Rates (hallucination, override, drift, policy violations) should stay low; accuracies and success rates (citation, retrieval, eval pass, tool-call) should stay high. Enter what your monitoring actually reports, not a target.
- Threshold direction & limits (fixed here)
- Whether higher or lower is worse, and the warn/critical thresholds, come from the signal catalog. They are a defensible starting point; in your own system you would calibrate them to the baseline the system was qualified against.
Monitor → detect → assess: what state is this AI system in?
Enter the current monitoring readings and SPEQ returns the assurance state and the action it triggers. A safety-, integrity-, or policy-critical breach suspends the system; any other breach requires reassessment; a warning raises a monitoring alert. Deterministic — thresholds are a starting operating point you set from your own qualified baseline.
All monitored signals are within limits — continue monitoring at the defined cadence.
HOW TO READ THE OUTPUT
- The state is set by the single worst signal, not an average. One suspend-on-breach signal in breach — a spiking hallucination rate, a collapsed citation accuracy — suspends the system even if everything else is green, because averaging away a safety signal is exactly the failure mode to avoid.
- Monitoring-alert is an early-warning, not a pass. A warning-threshold crossing means a trend is heading the wrong way; the cheap action now (investigate, tighten monitoring) is what prevents a breach later.
- Reassessment-required maps straight to the change-control ladder: rerun the affected evaluation cases and confirm the acceptance criteria still hold before continuing to rely on the system.
- Suspended means stop relying on the output, not necessarily stop the service. It is the point where continued use is no longer defensible without human approval to return the system to service after reassessment.
WORKED EXAMPLE
A deployed regulatory copilot: retrieval accuracy has slipped to 0.88 (warning) and the hallucination rate has risen to 0.06, above its 0.05 critical threshold.
- Hallucination rate
- 0.06 (critical 0.05)
- Retrieval accuracy
- 0.88 (warn 0.90)
- Other signals
- within limits
RESULT
The retrieval warning alone would be a monitoring alert, but the hallucination rate crossing its critical threshold is a suspend-on-breach signal, so the system is suspended: reliance on its output is no longer defensible until the cause is found, the evaluation is rerun, and a human approves its return to service. The retrieval slip is a second thread to investigate, not the reason for the suspension.
REGULATORY BASIS
- FDA/EMA — Guiding Principles of Good AI Practice in Drug Development (Jan 2026)
- Frames lifecycle monitoring and the detection of performance change for AI in regulated development; see SPEQ’s Good AI Practice page.
- ISPE GAMP 5 (2nd ed.) — A Risk-Based Approach to Compliant GxP Computerized Systems
- The ongoing-operation, monitoring, and periodic-review expectations for computerized systems this maps onto AI signals.
- ISO/IEC 42001:2023 — AI management systems
- The management-system requirement to monitor AI performance and act on deviations across the lifecycle.