AI Risk Assessment: Controls to Review Before Production

Is your AI system actually ready for production, or does it just work in a demo?

A working demo is not release evidence. A system can answer test questions correctly, and still have no owner for failures, no protections against malicious inputs, no threshold for monitoring, no reliable rollback path.

A pre-production AI risk assessment should determine the inherent risk, test the controls that reduce it, rate the remaining residual risk and turn that evidence into a release decision. The result should be a go, conditional go, or no-go with rationale, owners and conditions recorded.

This review is for a near production release candidate. It is not a substitute for an early AI readiness assessment or the full AI governance framework. It validates the deployed design and operating plan against the controls required for this workload.

DIRECT ANSWER: The assessment should answer seven questions: What can go wrong? How severe could it be? How plausible is it? Which controls reduce the risk? What evidence proves those controls work? What risk remains? Who has authority to accept or block release?

AI risk assessment at a glance

Step	Question	Output
1. Scope	What exact release candidate is being reviewed?	System boundary, version, users, data, tools, vendors and jurisdictions
2. Identify risk	What are the worst credible failure scenarios?	Risk statements tied to affected parties and business outcomes
3. Rate inherent risk	How severe and plausible is each scenario before controls?	Impact and likelihood rating
4. Map controls	What prevents, detects, responds to or recovers from the failure?	Control-to-evidence matrix
5. Test evidence	Did the controls work in representative and adversarial conditions?	Versioned test package and exceptions
6. Rate residual risk	What remains after validated controls?	Residual risk, owner and treatment
7. Decide release	Does the residual risk meet release criteria?	Go, conditional go or no-go record

Set scope, context and impact level

Instead of starting with a generic model, start with the release candidate. Identify the exact model and version, prompts or features, retrieval sources, tools, integrations, user interface, deployment environment and support model. Note what’s different since last review.

Describe the users, affected parties, decisions, and actions. State jurisdictions, data classes, scale and availability requirements. Then write out the worst credible outcomes across customers, employees, finances, operations, safety, rights and reputation.

The depth of evaluation should be driven by impact and autonomy. The risk profile of a drafting assistant whose output is reviewed before use is different than that of an agent who can approve a refund or update an account. Reversibility and contestability are important. If the person affected cannot detect or challenge an error, then more robust controls may be needed.

The NIST AI Risk Management Framework provides a useful structure for connecting context, measurement, accountability and treatment. Use the laws, standards and internal policies that apply to the organization’s sector and jurisdictions alongside it.

Rate inherent risk before evaluating controls

A control review becomes a risk assessment only when the team states how serious a failure could be and how plausible it is before treatment. Use a simple rating method that reviewers can apply consistently. The method below uses a 1 to 5 impact scale and a 1 to 5 likelihood scale.

Rating	Impact	Likelihood
1	Negligible effect, easily corrected	Rare under expected operating conditions
2	Limited effect on a small group or workflow	Unlikely but credible
3	Material operational, financial or user impact	Possible in normal use or foreseeable misuse
4	Major loss, rights impact, prolonged disruption or significant exposure	Likely without stronger controls
5	Severe or systemic harm, safety impact, large-scale rights or business consequence	Frequent or expected without treatment

Inherent risk score = impact × likelihood. Use the score as a routing aid rather than a substitute for judgement. A critical legal, safety, security or rights condition can block release even when arithmetic produces a moderate total.

Score	Band	Typical treatment
1-4	Low	Standard controls and owner review
5-9	Moderate	Targeted testing and named treatment
10-16	High	Independent review, deeper testing and explicit residual-risk approval
17-25	Critical	Do not release until risk is reduced or scope changes

After controls are tested, re-rate the scenario to establish residual risk. Record whether each residual risk is accepted, mitigated, transferred or unresolved, and name the person authorized to make that decision.

Create a control-to-evidence matrix

Create the matrix at the start of the review. Each row should identify the risk, required control, evidence owner, reviewer, acceptance criterion and status. This prevents important findings from disappearing inside meeting notes and gives the final approver a direct view of what passed, failed or remains conditional.

Risk scenario	Control	Evidence	Acceptance criterion	Owner	Status
Indirect prompt injection causes unauthorized tool action	Untrusted-content isolation, tool allowlist and scoped identity	Adversarial test set and action logs	No unauthorized action across defined attack cases	AI security lead	Open
RAG exposes restricted documents	Source-level authorization and revocation	Permission tests across user roles	Zero unauthorized retrieval in test suite	Data owner	Open
Unsupported answer is presented as factual	Evidence threshold, refusal path and citation validation	Evaluation results by question type	Meets defined factual-support threshold	Product owner	Open
Monitoring misses a critical failure	Alert coverage and tested escalation	Simulated threshold breach	Alert reaches named responder within target	Service owner	Open
Vendor model change alters behaviour	Version pinning or change notice plus reassessment trigger	Contract and change-control test	Material changes cannot reach production without review	Vendor owner	Open

Review data and retrieval controls

The data review should include training or fine-tuning data where relevant, application inputs, prompts, uploaded documents, retrieval sources, embeddings, outputs, feedback and logs. Access, deletion and retention, location, sensitivity, permitted uses, rights Source to processing to storage trace data.

Compare quality to the workload. Identify missing values, stale sources, duplicates, conflicting records, representation gaps, and upstream definition changes. Date and acceptance criteria. Record sample, method.

For retrieval-augmented generation, test more than answer quality:

• Source coverage for the questions users will ask

• Retrieval of authoritative and current material

• Enforcement of document-level access restrictions

• Behaviour when sources conflict or no support exists

• Citation accuracy and support for the generated claim

• Deletion, revocation and index-refresh timing

Include adversarial or irrelevant documents. A retrieval pipeline may generate fluent output from the wrong source, and a surface review may not catch the failure. Separate independent evaluation data from prompt tuning and model selection where impact warrants.

Review model behaviour and evaluation evidence

Evaluation should represent the production task, users and failure conditions. A provider benchmark can inform model selection, but it cannot approve the enterprise workflow.

Create a versioned evaluation package with the model, configuration, system prompt, tools, retrieval snapshot, evaluation data, metrics, thresholds, test date and reviewer sign-off. Preserve enough information to repeat the test after a material change.

Measure task quality and material harms. Depending on the workload, this can include factual support, classification performance, extraction accuracy, harmful output, bias, refusal behaviour, uncertainty, action correctness and escalation. Segment results where averages could hide a weak subgroup or rare critical failure.

The NIST Generative AI Profile identifies risks that can be unique to or intensified by generative AI and provides actions across the lifecycle. Its guidance on pre-deployment testing, content provenance and incident disclosure can inform the evidence package.

Review security and abuse cases

Threat modelling should cover the full application. The model is one component inside identities, data flows, tools, code, services and vendors.

Test prompt injection through direct user input and indirect content retrieved from documents, websites or messages. Verify that untrusted instructions cannot override system goals, expose protected data or trigger unauthorized tools.

For agents, test goal hijacking, tool misuse, privilege escalation, identity abuse, memory poisoning, insecure inter-agent communication and cascading failures. The OWASP Top 10 for Agentic Applications 2026 provides a current threat reference. MITRE ATLAS can help teams translate adversary tactics into prevention, detection, response and recovery scenarios.

Confirm least privilege, credential separation, allowlisted actions, input and output validation, transaction limits, network boundaries, secrets handling and rate controls. Test controls in combination because filtering, model safeguards and output validation can each appear effective alone and still fail across a chained workflow.

Review human oversight and user experience

Human review is effective only when the person has authority, time and useful context. Identify which actions require approval, what the reviewer sees and how disagreement is recorded.

Test the interface. Can the reviewer see source material, model uncertainty, changed records and the consequence of approving? Can the reviewer edit, reject, escalate or request more information? Does time pressure encourage automatic acceptance?

Define disclosures appropriate to users and affected parties. For consequential decisions, confirm how a person can question or contest an outcome and how the organization investigates it. Design for safe failure when the model is unavailable, uncertain or outside scope.

Canadian federal departments should consult the Algorithmic Impact Assessment and Directive on Automated Decision-Making for systems within their scope. Other organizations should identify their own statutory, regulatory and contractual duties.

Review operations, monitoring and rollback

You need an operating plan for production approval. Retrieval, Model and Prompt Changes, Data Pipelines, Integrations, Access, Infrastructure, Evaluation, Incidents and Vendor Management. Name Owners.

Monitoring should target system health and quality of outcome. Signals can be latency, error rate, tool failures, retrieval misses, policy violations, harmful outputs, drift, unusual usage, cost and user escalation. Set thresholds, owners and what to do Logging should facilitate investigation without collecting data beyond legitimate need.

Runbooks should include containment, stakeholder communication, evidence preservation, fallback and recovery. Test at least one representative failure in a tabletop or controlled exercise. Check log coverage and alert delivery. Simulate a breach of threshold and follow it through from detection to acknowledgment and response.

Rollback can mean reverting a model, prompt, data index, tool permission, application release or vendor endpoint. Confirm that dependencies are versioned and that operators can restore a known state. Where internal coverage is insufficient, define the role of managed AI services before release.

Record residual risk and the release decision

A control review will not remove every risk. Record what remains after treatment and classify each item as accepted, mitigated, transferred or unresolved. Name the owner and approver.

Release record field	What to capture
System and version	Exact release candidate reviewed
Scope and impact	Users, actions, jurisdictions, impact level and applicable requirements
Evidence	Tests examined and acceptance thresholds
Control exceptions	Failed, waived or conditional controls
Residual risk	Remaining risk, owner and treatment
Decision	Go, conditional go or no-go, approver and date
Conditions	Actions, owners, target dates and restrictions
Reassessment triggers	Changes that require targeted testing or new approval

A go decision means required evidence meets release criteria.

A conditional go includes specific actions, owners, target dates and restrictions.

A no-go identifies the blocking evidence or control and the route to reassessment.

Do not use conditional approval to move unresolved critical risk into production without a credible constraint.

Representative release decision

Consider a customer-support RAG assistant that retrieves policy documents and drafts responses for human approval. Evaluation quality meets the target, document-level permissions pass, and the human reviewer can see sources. A monitoring alert has been configured but never tested end to end.

Finding	Risk result	Treatment	Release effect
Answer quality meets threshold	Residual risk low	Continue monitoring	Pass
Restricted-source retrieval test passes	Residual risk low	Re-test after index or identity change	Pass
Monitoring alert untested	Residual risk moderate	Simulate threshold breach before broad release	Conditional
Human review required before customer send	Impact reduced by approval boundary	Keep reviewer context and audit log	Pass

A conditional go can be reasonable if the release is limited, the monitoring test has a named owner and near-term deadline, and the person with stop authority can enforce the condition. If the untested control were the only protection against unauthorized tool actions, the correct decision could be no-go instead.

Cylix in practice: traceability and human control

The Kirby Structures case study describes role-based access, encrypted communications, audit logging, secure model configuration, source-level traceability and human commercial review around AI-supported estimation. Those controls illustrate why production risk needs to be assessed at the workflow level rather than inferred from model quality alone.

Approval should be traceable

An AI risk assessment is useful when reviewers can trace each material risk to evidence, an owner and a release decision. Set the context, rate inherent risk, inspect data and retrieval, evaluate behaviour, test security, validate human oversight and prove the operating response. Then record residual risk and approval in a form that can be revisited after change or incident.

NEXT STEP: Book an AI Risk Workshop to review one release candidate against its current data, tools, integrations, infrastructure, threat model and operating plan.