AI Testing and Quality Assurance
One of the thirty-eight critical capabilities, shown as it is written and used. TQA sits in MC4, AI Delivery and Operations.
- 16building blocks
- 4categories
- 5levels per block
- 64improvement actions
- 15management artefacts
Definition and purpose
AI Testing and Quality Assurance is the capability to verify, independently of those who built an AI system, that it performs as specified, behaves safely and fairly, resists misuse and meets its regulatory obligations, before release and after every material change.
AI systems cannot be verified the way conventional software is. Their behaviour is learned from data, often non-deterministic, and defined over a continuous input space, so exhaustive coverage is impossible. Testing therefore aims at sufficient, documented confidence: representative sampling, statistical validation, boundary and stress testing, and adversarial probing that together show how the system behaves and where it fails. A central difficulty, often called the test oracle problem, is deciding what the correct output is at all.
The capability extends quality assurance beyond function into safety, fairness, adversarial resilience and compliance. For generative and agentic systems it also covers output quality, such as accuracy, groundedness and task success, and unsafe actions, not only incorrect predictions.
| Goal | What it means in practice |
|---|---|
| Risk-based strategy and standards | Testing depth and minimum requirements follow each system's risk, and are enforced at lifecycle gates |
| Independent verification | Quality sign-off comes from people independent of the build team, with authority to block release |
| The full quality envelope | Functional performance, robustness, integration, regression, user acceptance, safety, fairness, adversarial resilience and compliance are all tested where relevant |
| Quality at delivery speed | Automated tests act as gates on every model, code and data change |
| Learning from defects | Defects are classified with an AI-specific taxonomy, root causes are addressed, and escapes improve the testing itself |
| Governance-grade evidence | Test records are a standard deliverable, sufficient for governance, audit and regulatory review |
Scope and boundaries
This capability owns independent testing of AI systems. It does not own the developer's own evaluation, production monitoring, or the requirements that testing checks against. Stating the boundary once means no practice is assessed twice.
| Practice | Owned by | The testing capability's role |
|---|---|---|
| Developer evaluation while models are built | AI Model Development and Engineering (MDE) | Receives the results; tests independently |
| Generative AI evaluation methods and criteria | Generative AI and Foundation Model Management (GFM) | Applies them in independent testing |
| Agent safety test requirements | Agentic AI and Autonomous Systems (AAS) | Runs independent agent safety and adversarial tests |
| Fairness and safety test standards | Responsible AI and Ethics Management (RAE) | Implements them in safety and fairness testing |
| Regulatory requirements to test against | AI Risk and Compliance Management (ARC); AI Regulatory Intelligence (ARI) | Maps them to tests and produces the evidence |
| Delivery pipelines | MLOps and Model Lifecycle Management (MLO) | Supplies test stages that run as pipeline gates |
| Production monitoring and drift | AI Performance Monitoring and Optimisation (PMO) | Supplies baselines; retests before redeployment |
| Security testing of infrastructure | AI Security and Data Protection (ASD) | Covers AI-specific adversarial testing only |
Relationships and dependencies
| Capability | Relationship | What passes between them |
|---|---|---|
| MDE | Provides input to TQA | Trained models, model cards and developer evaluation results |
| UCA | Provides input to TQA | Success criteria, thresholds and testability requirements; TQA returns design-quality findings |
| RAE | Governs TQA | Fairness, safety and explainability test standards; receives independent assurance results |
| ARC | Governs TQA | Regulatory and risk-based test requirements; receives compliance evidence |
| GFM | Shares practice with TQA | Methods for evaluating generative AI output |
| AAS | Shares practice with TQA | Agent safety testing |
| SDI | Depends on TQA | Cannot deploy without testing sign-off |
| PMO | Depends on TQA | Uses testing baselines for monitoring; triggers retesting |
| KMS | Depends on TQA | Captures test methods, adversarial libraries and defect patterns |
| EIM | Depends on TQA | Uses lightweight test standards for experiments and full standards before production |
Prerequisites. Independent performance testing (B-05) cannot exceed Level 2 until use case design supplies evaluation criteria. Fairness testing (C-11) and safety testing (C-10) cannot exceed Level 2 until Responsible AI has set the test standards they apply.
The sixteen building blocks
The capability is assessed through sixteen building blocks in four categories. Each carries a one-line statement of what is assessed.
| ID | Building block | What is assessed |
|---|---|---|
| A Testing strategy and framework | ||
| TQA-A-01 | AI Test Strategy | Whether testing scope and depth are set by risk and applied across the portfolio |
| TQA-A-02 | AI QA Standards and Gate Criteria | Whether minimum test requirements are enforced at lifecycle gates |
| TQA-A-03 | Testing Roles, Independence and Accountability | Whether sign-off is independent and able to block release |
| TQA-A-04 | Test Environment and Test Data Management | Whether test conditions and data are representative of production and controlled |
| B Functional and performance testing | ||
| TQA-B-05 | Model Performance and Output Quality Testing | Whether performance, and output quality for generative systems, is independently verified against thresholds |
| TQA-B-06 | Robustness and Stress Testing | Whether behaviour under noise, shift and load is characterised |
| TQA-B-07 | Integration and System Testing | Whether end-to-end data flows, contracts and fallbacks are verified |
| TQA-B-08 | Regression Testing | Whether changes are automatically checked for unintended degradation |
| TQA-B-09 | User Acceptance Testing | Whether real users confirm usefulness and appropriate reliance |
| C Safety, fairness and compliance testing | ||
| TQA-C-10 | AI Safety Testing | Whether failure modes, fallbacks and harmful outputs or actions are tested |
| TQA-C-11 | Fairness and Bias Testing | Whether fairness is independently tested across groups |
| TQA-C-12 | Red-Teaming and Adversarial Testing | Whether deliberate misuse and attack are tested |
| TQA-C-13 | Regulatory and Compliance Testing | Whether regulatory requirements are mapped to tests with audit-grade evidence |
| D Automation, defects and improvement | ||
| TQA-D-14 | Test Automation | Whether tests run automatically as pipeline gates |
| TQA-D-15 | Defect Management | Whether defects are classified, root-caused and prevented |
| TQA-D-16 | Testing Continuous Improvement | Whether testing effectiveness is measured and improved |
Maturity profile and targets
This summarises typical patterns only. An organisation's actual profile is its set of building-block levels, which will usually be uneven.
| Level | A typical organisation at this level |
|---|---|
| 1 Initial | Developers test their own systems with conventional methods. There is no AI test strategy, no standards and no independent sign-off. Fairness, safety and adversarial testing are absent, and quality problems are found in production. |
| 2 Basic | Some AI-specific testing exists on important projects, such as structured performance checks and occasional fairness reviews. Standards are drafts; independence and automation are partial. Coverage depends on the project. |
| 3 Intermediate | An approved strategy and gate criteria apply to all in-scope systems. Independent testers sign off. Performance, integration, regression, safety, fairness, adversarial and compliance testing are defined, and records show they are used. |
| 4 Advanced | Testing depth follows risk tier. Results feed monitoring, safeguards and governance decisions. Automation gates every change. Defect patterns and effectiveness metrics are tracked, and escaped-defect rates fall. |
| 5 Optimising | Testing-effectiveness data drives documented improvement cycles. Methods are updated ahead of new system types and threats, and the gap between tested and observed production behaviour is tracked and narrowed. |
Target levels by context
The scale is the same for everyone; the target depends on what the AI is used for. These targets are indicative and are agreed with each organisation during an assessment.
| Context | Indicative target |
|---|---|
| Low-risk internal AI use in a smaller organisation | Level 3 on A-01 to A-03, B-05, B-07 and B-08; Level 2 acceptable elsewhere |
| Customer-facing or public-sector AI | Level 3 on every building block; Level 4 on A-03, C-10 and C-11 |
| High-risk or regulated AI | Level 4 on A-02, A-03, B-05, C-10 to C-13 and D-15; Level 3 minimum elsewhere |
Building-block level descriptors
Every building block is described on all five levels, with the tests that separate the hardest boundaries and the evidence that would demonstrate each level. Open any block to read it in full.
A Testing strategy and framework
TQA-A-01AI Test Strategy
Whether testing scope and depth are set by risk and applied across the portfolio.
- 1 InitialNo AI test strategy. Each project decides its own testing, usually with conventional software methods.
- 2 BasicA general testing approach is used on some AI projects but does not address AI-specific dimensions or vary depth by risk.
- 3 IntermediateAn approved AI test strategy defines scope, AI-specific test dimensions, risk-based depth and lifecycle stages, and is applied to all in-scope systems.
- 4 AdvancedTest depth is set per system from its risk tier; the strategy is reviewed on a defined cycle and coverage by tier is reported to governance.
- 5 OptimisingThe strategy is revised using testing-effectiveness and incident data, and the effect of each revision is measured.
- Tests between levels
- 2→3 sampled test plans cite the strategy for every in-scope system.
3→4 test depth demonstrably differs by risk tier.
4→5 strategy changes trace to effectiveness data. - Typical evidence
- Documents: strategy with approval and version history, risk-tier-to-depth mapping. Records: sampled test plans, review minutes.
TQA-A-02AI QA Standards and Gate Criteria
Whether minimum test requirements are enforced at lifecycle gates.
- 1 InitialNo AI QA standards; release quality is negotiated case by case.
- 2 BasicQuality guidelines exist but are optional or inconsistently applied; no minimum requirements.
- 3 IntermediateMinimum testing requirements and thresholds (performance, fairness, safety, documentation) are enforced at lifecycle gates; exceptions are recorded and approved.
- 4 AdvancedStandards are differentiated by risk tier; gate outcomes and exception rates are monitored, and standards are updated on a defined cycle.
- 5 OptimisingStandards are revised from gate outcome and escaped-defect data, and the effect on production quality is measured.
- Tests between levels
- 2→3 gate records show at least one enforced block or approved exception.
3→4 tiered thresholds, plus exception-trend reporting. - Typical evidence
- Documents: standards catalogue. Records: gate decision records, exception register. Metrics: exception rate by tier.
TQA-A-03Testing Roles, Independence and Accountability
Whether sign-off is independent and able to block release.
- 1 InitialNo defined testing roles; developers self-certify.
- 2 BasicTesting roles exist on some projects, but independence and sign-off accountability are not defined.
- 3 IntermediateRoles, sign-off authority and independence requirements are defined. Sign-off for in-scope systems is given by someone independent of the build team, with authority to block.
- 4 AdvancedIndependence is structural and blocking decisions have been exercised and upheld. Testers are assessed on quality outcomes, not delivery speed.
- 5 OptimisingIndependence arrangements are reviewed against evidence, such as issues found independently versus by developers, and adjusted.
- Tests between levels
- 2→3 the sign-off signatory differs from the build team in sampled releases.
3→4 at least one upheld block, plus a separate reporting line or equivalent. - Typical evidence
- Documents: RACI. Records: sign-off records, blocked-release records. Interviews: testers and managers.
- Proportionality
- Smaller organisations may achieve independence through another team or an external reviewer.
TQA-A-04Test Environment and Test Data Management
Whether test conditions and data are representative of production and controlled.
- 1 InitialAd hoc environments and data that do not represent production.
- 2 BasicManaged environments or test data exist for some projects; representativeness is not checked.
- 3 IntermediateEnvironments and test data are provisioned through a defined process; representativeness against production is checked before testing, and access to test data is controlled.
- 4 AdvancedRepresentativeness is measured, for example by comparing test and production distributions; environment-caused test failures are tracked; costs are managed.
- 5 OptimisingEnvironment and data practices are improved from analysis of defects traced to environment or data gaps.
- Typical evidence
- Documents: environment plan. Records: representativeness checks, test data lineage and access logs. Metrics: environment-caused failure rate.
B Functional and performance testing
TQA-B-05Model Performance and Output Quality Testing
Whether performance, and output quality for generative systems, is independently verified against thresholds.
- 1 InitialNo independent performance testing; only the developer's evaluation.
- 2 BasicSome independent checks, but without held-out data, pass/fail thresholds or records.
- 3 IntermediateIndependent testing uses held-out data against defined thresholds with a stated confidence level. For generative systems, output quality (e.g. accuracy, groundedness, task success) is evaluated against documented rubrics by human or automated evaluators.
- 4 AdvancedIndependent results are compared with developer evaluation and differences investigated. Automated evaluators are validated against human judgement. Baselines are handed to PMO.
- 5 OptimisingThe gap between tested and observed production performance is tracked, and evaluation methods are changed to narrow it.
- Tests between levels
- 2→3 held-out data is not accessible to the build team; thresholds are defined before testing.
3→4 records show developer–tester reconciliation and evaluator validation. - Typical evidence
- Records: test reports, held-out data access controls. Documents: evaluation rubrics. Metrics: human–automated evaluator agreement.
TQA-B-06Robustness and Stress Testing
Whether behaviour under noise, shift and load is characterised.
- 1 InitialNo robustness testing; failure modes are unknown at release.
- 2 BasicAd hoc robustness checks on high-visibility systems only.
- 3 IntermediateDefined tests cover noisy and missing data, distribution-shift scenarios and load, producing a documented operating envelope for each in-scope system.
- 4 AdvancedEnvelope findings set monitoring thresholds and deployment safeguards, with traceable links to PMO and SDI.
- 5 OptimisingEnvelope predictions are compared with production incidents, and scenarios are updated.
- Typical evidence
- Records: robustness reports, links to monitoring thresholds. Documents: operating-envelope statements.
TQA-B-07Integration and System Testing
Whether end-to-end data flows, contracts and fallbacks are verified.
- 1 InitialNo structured integration testing; issues surface in production.
- 2 BasicSome checks on high-risk integrations; no end-to-end testing.
- 3 IntermediateEnd-to-end tests verify data flows, interface contracts, error propagation and fallbacks before every release.
- 4 AdvancedContract tests are automated; integration-related production incidents are tracked and falling.
- 5 OptimisingTest design is updated from integration incident root causes.
- Typical evidence
- Records: system test reports, automated contract test runs. Metrics: integration incident trend.
TQA-B-08Regression Testing
Whether changes are automatically checked for unintended degradation.
- 1 InitialNo regression testing.
- 2 BasicManual regression checks for major releases only.
- 3 IntermediateAutomated regression suites run on model, code and data changes, covering performance and fairness metrics, with thresholds that block release.
- 4 AdvancedSuites are tiered by execution time; flakiness and coverage are measured and controlled.
- 5 OptimisingSuites are improved from regressions that escaped to production.
- Typical evidence
- Records: pipeline run history. Documents: suite design. Metrics: flakiness rate, escaped regressions.
TQA-B-09User Acceptance Testing
Whether real users confirm usefulness and appropriate reliance.
- 1 InitialNo structured UAT.
- 2 BasicSome user testing on high-visibility products, without defined criteria.
- 3 IntermediateStructured UAT with representative users and realistic tasks, with sign-off criteria covering usefulness and whether users understand the system's limitations (trust calibration).
- 4 AdvancedUAT findings are compared with post-deployment adoption and reliance data.
- 5 OptimisingUAT methods are adjusted based on that comparison.
- Typical evidence
- Documents: UAT plans. Records: UAT results and sign-offs. Metrics: adoption and override rates after release.
C Safety, fairness and compliance testing
TQA-C-10AI Safety Testing
Whether failure modes, fallbacks and harmful outputs or actions are tested.
- 1 InitialNo safety testing; fallbacks and error handling are unverified.
- 2 BasicSome safety checks on high-risk systems, without systematic scenarios.
- 3 IntermediateDefined safety testing covers fault injection, boundary conditions, fallback and graceful degradation, and harmful outputs. For agentic systems it also covers unsafe actions and tool misuse. It applies to all systems above a set risk threshold.
- 4 AdvancedEvery safety finding traces to a safeguard decision; all systems above the threshold are covered.
- 5 OptimisingScenarios are updated from incidents and near-misses, and scenario yield is measured.
- Tests between levels
- 2→3 a scenario catalogue exists and is applied to every system above threshold.
3→4 finding-to-safeguard traceability. - Typical evidence
- Documents: scenario catalogue. Records: safety test reports, safeguard decision records.
TQA-C-11Fairness and Bias Testing
Whether fairness is independently tested across groups.
- 1 InitialNo independent fairness testing.
- 2 BasicSome fairness checks, but not independent and without defined group coverage.
- 3 IntermediateIndependent testing covers defined groups and, where data permit, intersections. Reports include contextual analysis and a documented determination.
- 4 AdvancedFindings drive governance decisions; mitigations are retested before release.
- 5 OptimisingMethods are improved from post-deployment fairness monitoring.
- Tests between levels
- 2→3 the tester is independent of the build team and group coverage is defined in advance.
- Typical evidence
- Records: fairness test reports, governance decisions citing them, mitigation retests.
- Proportionality
- Whether protected-attribute data may be collected and used varies by jurisdiction. Where it cannot be, proxy or sampling methods with documented limitations are acceptable.
TQA-C-12Red-Teaming and Adversarial Testing
Whether deliberate misuse and attack are tested.
- 1 InitialNo adversarial testing.
- 2 BasicInformal adversarial checks on high-visibility systems.
- 3 IntermediateStructured adversarial testing of risk-relevant systems against a maintained threat catalogue, e.g. prompt injection, jailbreaks, evasion, data poisoning and agent tool misuse, by testers with an adversarial brief.
- 4 AdvancedFindings are tracked to remediation and retest; the catalogue is updated from threat intelligence.
- 5 OptimisingScope and methods are adjusted from measured finding yield and incident data.
- Typical evidence
- Documents: threat catalogue. Records: red-team reports, remediation and retest records.
TQA-C-13Regulatory and Compliance Testing
Whether regulatory requirements are mapped to tests with audit-grade evidence.
- 1 InitialNo compliance testing; evidence cannot be produced on request.
- 2 BasicAd hoc checks for some regulated systems; evidence assembled when asked.
- 3 IntermediateApplicable requirements are mapped to tests, and evidence packages are a standard deliverable for every regulated system.
- 4 AdvancedEvidence packages pass internal audit or regulatory review without rework; mappings update when ARC or ARI flag a change.
- 5 OptimisingEvidence design is improved from audit and review findings.
- Tests between levels
- 3→4 audit or review outcomes show no material evidence gaps.
- Typical evidence
- Documents: requirement-to-test mapping. Records: evidence packages, audit findings.
D Automation, defects and improvement
TQA-D-14Test Automation
Whether tests run automatically as pipeline gates.
- 1 InitialAll testing is manual.
- 2 BasicSome automated scripts exist, outside the delivery pipeline.
- 3 IntermediateData validation, model evaluation, integration and regression tests run automatically as pipeline gates.
- 4 AdvancedAutomated coverage, flakiness and gate execution time are measured and kept within agreed limits.
- 5 OptimisingAutomation investment is directed by analysis of manual effort and escaped defects.
- Typical evidence
- Records: pipeline configuration and run history. Metrics: coverage and flakiness.
TQA-D-15Defect Management
Whether defects are classified, root-caused and prevented.
- 1 InitialDefects tracked informally, if at all.
- 2 BasicDefects are tracked, but without an AI-specific taxonomy or root cause analysis.
- 3 IntermediateAn AI-specific taxonomy (e.g. data, model, integration, fairness, safety), severity rules, and root cause analysis for significant defects are applied.
- 4 AdvancedDefect patterns are fed back to MDE and UCA; the recurrence rate is tracked and falling.
- 5 OptimisingPrevention measures are evaluated for their effect and adjusted.
- Typical evidence
- Documents: taxonomy. Records: defect log and RCA records. Metrics: recurrence rate.
TQA-D-16Testing Continuous Improvement
Whether testing effectiveness is measured and improved.
- 1 InitialNo improvement process; the same gaps recur.
- 2 BasicImprovements happen after notable incidents only.
- 3 IntermediateA defined process reviews escaped defects and effectiveness metrics and maintains a testing improvement plan.
- 4 AdvancedEffectiveness metrics (e.g. escape rate, stage of detection) show sustained improvement.
- 5 OptimisingNew system types and threats are brought into scope before they reach production, and the improvement process itself is reviewed.
- Typical evidence
- Records: escaped-defect reviews. Documents: improvement plan. Metrics: escape rate trend. ## 7. Practices, outcomes and metrics (POM catalogue).
Practices, outcomes and metrics
The practice catalogue lists each practice, the outcome it should produce and how to measure it. Tags show the building blocks and levels each practice evidences, so the measures in an improvement plan match the gaps it addresses. Metric types distinguish activity, capability, outcome, value and risk measures. An extract:
| ID | Blocks and levels | Practice | Expected outcome | Metric | Type |
|---|---|---|---|---|---|
| TQA-P-01 | A-01, A-02 L3 | Apply the risk-tiered test strategy and gate criteria to every in-scope system | Consistent minimum quality at release | % of in-scope releases passing gates without undocumented exceptions | Capability |
| TQA-P-03 | A-03 L3–4 | Independent sign-off with authority to block | Credible assurance for governance | % of releases signed off by an independent tester | Capability |
| TQA-P-05 | B-05 L3–5 | Independent testing on held-out data; validated evaluators for generative AI | Performance claims can be trusted | Gap between tested and first-90-day production performance | Outcome |
| TQA-P-08 | B-08, D-14 L3–4 | Automated regression gates on model, code and data changes | Safe, fast iteration | Regressions escaping to production; gate flakiness rate | Risk |
| TQA-P-10 | C-11 L3–4 | Independent fairness testing with contextual analysis | Fair outcomes and defensible decisions | Disparity findings resolved before release; post-release disparity breaches | Risk |
| TQA-P-11 | C-12 L3–5 | Red-teaming against a maintained threat catalogue | Vulnerabilities found before attackers find them | Critical findings open past due date; findings per exercise | Risk |
| TQA-P-13 | D-15 L3–4 | AI defect taxonomy with root cause analysis | Defects do not recur | Recurrence rate of defect classes | Outcome |
| TQA-P-14 | D-16 L3–5 | Escaped-defect review and improvement plan | Testing gets better over time | Defect escape rate; share of defects found before production | Outcome |
Before a metric is used, it is given a definition, a data source and a baseline.
Improvement actions
The action catalogue holds at least one candidate action for every building block at every level step: 64 actions in all, so an improvement plan can address any gap an assessment reveals. Each action names its prerequisites, in this capability or others. Three blocks are shown in full:
| ID | Block and step | Action | Prerequisites |
|---|---|---|---|
| Independent sign-off (TQA-A-03) | |||
| TQA-X-09 | A-03 1→2 | Name a testing lead and define testing roles for AI projects | — |
| TQA-X-10 | A-03 2→3 | Define independence rules and blocking authority for sign-off | TQA-X-09; AGP decision rights |
| TQA-X-11 | A-03 3→4 | Separate reporting lines, or an equivalent; set quality-outcome objectives for testers | TQA-X-10 |
| TQA-X-12 | A-03 4→5 | Compare issues found independently with those found by developers; adjust arrangements | TQA-X-11 |
| Performance and output quality (TQA-B-05) | |||
| TQA-X-17 | B-05 1→2 | Introduce independent spot checks of model performance before release | — |
| TQA-X-18 | B-05 2→3 | Establish held-out test data and pre-set thresholds; define generative AI evaluation rubrics | TQA-X-14; UCA criteria; GFM methods |
| TQA-X-19 | B-05 3→4 | Reconcile independent and developer results; validate automated evaluators | TQA-X-18 |
| TQA-X-20 | B-05 4→5 | Track the tested-versus-production gap; revise evaluation methods | TQA-X-19; PMO |
| Fairness and bias testing (TQA-C-11) | |||
| TQA-X-41 | C-11 1→2 | Run fairness checks on high-impact systems | RAE fairness standards |
| TQA-X-42 | C-11 2→3 | Make fairness testing independent, with defined group coverage and contextual analysis | TQA-X-41; TQA-X-10 |
| TQA-X-43 | C-11 3→4 | Require governance decisions on findings; retest mitigations | TQA-X-42; AGP |
| TQA-X-44 | C-11 4→5 | Improve methods from post-deployment fairness monitoring | TQA-X-43; PMO |
Prerequisites cross blocks and capabilities. Independent fairness testing (TQA-X-42) depends on independence rules for sign-off (TQA-X-10), so the two are sequenced together in a plan. The improvement method shows how.
Typical challenges
| Challenge | Typical cause | Response | Blocks |
|---|---|---|---|
| Testing is a late-stage gate, so it detects problems rather than preventing them | The lifecycle is planned as build, then test | Build test activity into every lifecycle stage, starting from testability requirements in use case design | A-01, B-05 |
| Independence exists on the organisation chart but not in practice | Testers report to delivery managers and are measured on speed | A separate reporting line or equivalent; quality-outcome objectives; protected blocking authority | A-03 |
| Red-teaming is superficial | Testers apply a conventional QA mindset | An adversarial brief, external or rotated testers, and a threat catalogue with yield tracking | C-12 |
| Fairness reports give statistics without meaning | Measurement without contextual analysis | Add legal, operational and alternative-explanation analysis, and a clear determination | C-11 |
| Regression suites are too slow to gate every change | Single-tier suites that need full retraining | Tiered suites at commit, merge and release level | B-08, D-14 |
| User acceptance testing checks features but not reliance | A conventional UAT method | Expose users to realistic errors and add trust-calibration criteria | B-09 |
| Compliance evidence is assembled when an audit is announced | Evidence is treated as a by-product | A requirement-to-test mapping, and evidence packages as a standard deliverable | C-13 |
| Defect records do not support root cause analysis | A software bug taxonomy applied to AI | An AI-specific taxonomy: data, model, integration, fairness, safety, adversarial | D-15 |
Management artefacts
Each artefact is tagged as evidence for assessment, a management artefact, an improvement tool, or several of these. Every building block is covered by at least one artefact.
| ID | Artefact | Purpose | Type | Blocks |
|---|---|---|---|---|
| TQA-T-01 | AI Test Strategy | Scope, risk-based depth, test dimensions, lifecycle stages | Management, evidence | A-01 |
| TQA-T-02 | AI QA Standards Catalogue | Minimum requirements and gate thresholds by risk tier | Management, evidence | A-02 |
| TQA-T-03 | Testing RACI and Independence Statement | Roles, sign-off authority, independence rules | Management, evidence | A-03 |
| TQA-T-04 | Test Environment and Data Plan | Provisioning, representativeness, access control | Management, evidence | A-04 |
| TQA-T-05 | AI System Test Plan | Per-system scope, cases, data and criteria | Management, evidence | B-05 to C-13 |
| TQA-T-06 | Testing Results and Quality Report | Per-system results and release recommendation | Evidence | B-05 to B-09 |
| TQA-T-07 | Operating Envelope Statement | Documented behaviour limits and failure modes | Evidence, tool | B-06, C-10 |
| TQA-T-08 | AI UAT Plan and Report | UAT design, trust-calibration results, sign-off | Evidence | B-09 |
| TQA-T-09 | Safety Scenario Catalogue | Reusable safety and agentic scenarios | Tool, evidence | C-10 |
| TQA-T-10 | Fairness Testing Report | Groups, metrics, findings, context, determination | Evidence | C-11 |
| TQA-T-11 | Threat Catalogue and Red-Team Report | Threats tested, findings, remediation | Tool, evidence | C-12 |
| TQA-T-12 | Compliance Testing Evidence Package | Requirement mapping, tests, records, determination | Evidence | C-13 |
| TQA-T-13 | Test Automation Dashboard | Coverage, flakiness, gate time | Evidence, management | B-08, D-14 |
| TQA-T-14 | AI Defect Taxonomy and Log | Classification, severity, root cause, recurrence | Management, evidence | D-15 |
| TQA-T-15 | Testing Improvement Plan | Escaped-defect review outputs and planned changes | Tool, evidence | D-16 |
Skills
The skills this capability draws on, from the AI Skills Framework for AI-specific competence and SFIA 9 for professional testing and assurance skills.
| Framework | Skill | Blocks it supports |
|---|---|---|
| AI Skills Framework | AI Risk Management and Assurance | A-01, A-02, C-10 |
| AI Skills Framework | AI Fairness, Bias and Inclusivity | C-11 |
| AI Skills Framework | AI Audit, Transparency and Accountability | A-03, C-13 |
| AI Skills Framework | AI Governance, Policy and Compliance | A-02, C-13 |
| AI Skills Framework | AI/ML Model Engineering | B-05 to B-08 |
| SFIA 9 | Functional testing | B-05, B-07, B-08, B-09 |
| SFIA 9 | Non-functional testing | B-06, C-10 |
| SFIA 9 | Quality assurance | A-02, A-03, D-16 |
| SFIA 9 | Penetration testing | C-12 |
| SFIA 9 | Data science | B-05, C-11 |
References
- ISO/IEC TR 29119-11:2020, guidelines on the testing of AI-based systems
- ISO/IEC 25059:2023, quality model for AI systems
- ISO/IEC 42001:2023, AI management systems
- NIST AI Risk Management Framework (AI RMF 1.0) and its Generative AI Profile, NIST AI 600-1
- OWASP Top 10 for LLM Applications, and MITRE ATLAS, as sources for adversarial threat catalogues
- ISTQB Certified Tester AI Testing syllabus
Where to go next
See where your organisation stands on AI testing.
A free pilot assessment applies the full method to one capability, such as this one.