Capability maturity assessment

Before the assessment: the case for change

An assessment answers a question, so the question comes first. Before Stage 1, the sponsor and a small leadership group agree a short case for change: what needs to improve, why it matters now, and what a better future would look like in business terms. It names the goals the AI programme serves, which is what "importance" is later judged against.

The same conversation settles who will lead. Change of this kind needs more than one senior sponsor: it needs a small group of leaders with the authority, credibility and expertise to decide where effort goes and to find the resources. That group owns the assessment results and the choice of priority capabilities.

Two stages, from breadth to depth

Rating nearly five hundred building blocks in one pass would take too long and spread attention thinly. The assessment therefore runs in two stages: a broad look at every capability to find where it matters, then a deep look at the few that matter most.

Stage 1: all thirty-eight capabilities

Self-assessment. Respondents drawn from business, technology, risk and people functions rate each capability twice: its current maturity, against the capability maturity profile in its chapter, and its importance to the organisation's goals. Several respondents rate each capability, so the spread of answers shows where views differ.

Validation workshops and interviews. The self-assessment is tested in facilitated workshops, one per macro-capability, and in interviews with the owners of individual capabilities. Where respondents disagree, the workshop establishes why rather than averaging the answers. Ratings that survive validation are graded corroborated.

The map. The validated results are plotted as importance against maturity for all thirty-eight capabilities. The chart below is illustrative.

Choosing the priority capabilities. The deep dive covers the capabilities that are both important and immature, typically five to eight of them. A capability is a candidate when its importance is well above the organisation's middle score and its maturity is below its target. Two adjustments follow. A capability another candidate depends on can be added even if it sits outside the zone; in the example, ARC sits just outside the zone, but TQA's test strategy depends on ARC's risk tiers, so ARC could be added. And the final list is agreed with the sponsor, not produced by the chart alone.

The Stage 1 read-out also surfaces strategic themes: clusters of related capabilities that share a root cause, such as weak data foundations affecting several delivery capabilities at once.

Stage 2: the deep dive

Each priority capability is now assessed at building-block level. Where possible, the people who will later form that capability's improvement team take part, so the findings are theirs before the plan is written.

1. Deep-dive questionnaire. One item per building block, with the block's five level descriptors as the response options, plus "not applicable" and "don't know". Any answer above Level 2 names the evidence behind it.
2. Evidence. The artefacts and records the answers point to are collected. Where a descriptor refers to a population, such as every in-scope release, the assessor checks a sample.
3. Interviews and observation. Used where answers conflict or evidence is thin, and to trace root causes: why a block is where it is, not only where it is.
4. Skills gap analysis. The skills each chapter lists for its capability are compared with the skills the organisation has, so the plan can address people as well as practice.
5. Rating and calibration. The assessor applies the tests between levels and the evidence rule to each block. A calibration session checks a sample of ratings, and a second assessor independently rates a sample of blocks.

How a building block is rated

Three conditions decide a block's level.

• Levels are cumulative. A block is at Level 3 only if the descriptors for Levels 2 and 3 are both met.
• The test between levels is passed. Each chapter states what separates adjacent levels, for example "the sign-off signatory differs from the build team in sampled releases" for the step from Level 2 to Level 3 in independent testing sign-off.
• The evidence rule is met. Each level needs a minimum kind of evidence.

Where the evidence does not support the answer given, the block is rated at the highest level the evidence does support. Both the claimed and the evidenced ratings are recorded.

Each rating carries a confidence grade: claimed (answer only), corroborated (answer tested in workshops or interviews) or verified (an assessor has checked the evidence against the test between levels). Stage 1 ratings are corroborated; Stage 2 ratings aim to be verified.

From blocks to a capability result

A priority capability's full result is its profile of block levels. It also gets a headline level: the highest level that at least three-quarters of its applicable blocks have reached. An average would let strong blocks hide weak ones; a minimum would let a single block define the whole capability. The average, to one decimal place, is reported alongside for tracking change.

Prerequisite caps. Where a block has a prerequisite in another capability, its rating is capped until the prerequisite is met, and the cap is shown in the report. Independent performance testing, for instance, cannot be rated above Level 2 until use case design supplies the evaluation criteria to test against.

Worked example: AI Testing and Quality Assurance

TQA was one of the six priority capabilities in the illustrative map, self-rated at 1.7 in Stage 1. An illustrative deep-dive result for its sixteen building blocks of AI Testing and Quality Assurance:

• TQA-A-01: Level 3
• TQA-A-02: Level 3
• TQA-A-03: Level 2
• TQA-A-04: Level 2
• TQA-B-05: Level 3
• TQA-B-06: Level 2
• TQA-B-07: Level 3
• TQA-B-08: Level 3
• TQA-B-09: Level 2
• TQA-C-10: Level 2
• TQA-C-11: Level 2
• TQA-C-12: Level 1
• TQA-C-13: Level 2
• TQA-D-14: Level 3
• TQA-D-15: Level 2
• TQA-D-16: Level 2

Fifteen of the sixteen blocks are at Level 2 or above, but only six are at Level 3 or above. The headline level is therefore Level 2, and the average is 2.3, slightly better than the Stage 1 self-rating suggested. Against the indicative target for customer-facing AI (Level 3 on every block, Level 4 on A-03, C-10 and C-11), ten blocks are below target, four of them by two levels. Those gaps are what the improvement plan works from.

Practices shared between capabilities

Some practices appear in more than one capability. Incident management, for example, touches governance, risk, security, monitoring and agentic AI. Each shared practice has one primary owner, named in the scope and boundaries section of the owning chapter. The practice is rated there; the other capabilities refer to that rating rather than rating it again.

"Not applicable" and "don't know"

• Not applicable is agreed at scoping, with a reason. The capability or block is left out of the results and listed in the report, so readers can see what was excluded.
• Don't know triggers a follow-up with another respondent or an evidence request. It is never scored as Level 1 by default. If it remains unresolved, it is reported as unresolved.

What the reports contain

Consistency between assessors

Assessors calibrate on a shared set of worked cases before an engagement. During the deep dive, a second assessor independently rates a sample of blocks, and any difference of more than one level is reviewed and resolved before the report is issued. Descriptors or tests that produce repeated disagreement are flagged for revision in the framework itself.

Where to go next

• Capability Improvement Plan
  How the deep-dive results become a plan and a programme.
• One capability in full
  The descriptors and tests the deep dive uses.
• How the framework is built
  The architecture behind the assessment.