An AI Governance Framework for Models, Agents, Data, and Vendors
A model can pass evaluation and the system can still fail. The wrong document may enter retrieval. An agent may receive excessive permissions. A vendor may change a model without notice. A workflow may let users act on uncertain output without review.
An AI governance framework must cover the full system rather than the model alone. Enterprises need one control structure for four connected objects: models, agents, data and vendors.
Requirements should follow impact, autonomy and context, then continue across intake, design, validation, release, operation and retirement.
This is a structure you can use for predictive models, generative apps, and action-taking agents, without having to build a separate policy for each team. It brings governance into enterprise AI lifecycle where controls can affect architecture, testing, release and operations.
| THE FRAMEWORK: Govern models on purpose and evidence. Control Agents with Authority and Action. Data governance across source, use and retention Explicit shared responsibility for governing vendors. All four through one risk based lifecycle used. |
The AI governance framework at a glance
| Governance object | Primary question | Core controls | Evidence |
| Models | What is this model approved to do? | Intended use, evaluation, change control, monitoring, retirement | Versioned evaluation package, thresholds, change record |
| Agents | What authority can this agent exercise? | Scoped identity, tool limits, approval boundaries, memory controls, safe stop | Permission map, tool tests, action logs, red-team results |
| Data | What data may the system use and retain? | Rights, source controls, quality, lineage, retention, deletion, retrieval permissions | Data inventory, access tests, freshness and revocation results |
| Vendors | Which controls are performed by the supplier and which remain internal? | Shared responsibility, due diligence, change notice, audit evidence, exit | Contract terms, service evidence, version notices, export and deletion proof |
Start with principles, scope and risk appetite
Principles are only useful if they generate requirements. Fairness, transparency, privacy, security and accountability are all too broad to review individually. Translate each principle into observable evidence for the systems and decisions of the organization.
Define the scope in operational terms. Including in-house developed artificial intelligence systems, bought as products, accessed via public services, embedded in existing software or assembled from several components. Cover predictive models, generative applications, action taking agents. Include experiments if they use real sensitive data or impact real people.
Identify the accountable executive and the functions responsible for policy, portfolio oversight and system controls. Define the restricted uses and the prohibited uses. Prohibited use shouldn’t be able to go through an exception route. A restricted use can only proceed with stated conditions, evidence and approval.
Risk appetite should take into account impact and agency.” Inquire about the extent of financial, operational, legal, safety or rights-related exposure the enterprise will take, which actions require human approval, and what outcomes must stay reversible. Map those boundaries to a tiering model, with greater evidence for higher impact or more autonomous systems.
The NIST AI Risk Management Framework offers a voluntary structure to Govern, Map, Measure and Manage. Align those functions with the organization’s own policies, assurance processes and decision rights.
These governance requirements should be established within the enterprise AI roadmap, so teams can account for approval gates, technical controls and operational responsibilities before implementation begins.
Govern models by purpose and evidence
Direct answer: Model governance should define the approved purpose, versioned evidence, material-change rules, runtime monitoring and retirement path for each model in context.
Document the tasks the model supports, the contexts in which it was evaluated, known limitations and uses that remain outside approval. A model approved for summarizing internal documents is not automatically approved to recommend employment decisions.
Require versioned evaluation evidence that identifies the model, configuration, prompts or features, evaluation data, metrics, thresholds, date and reviewer. Test the end-to-end use case rather than relying on a provider benchmark. Production inputs, retrieval, tools and interface design can change behaviour.
Define material change before changes arrive. A new model version, fine-tune, system prompt, retrieval method, safety configuration or deployment location may trigger targeted testing or full reapproval. Monitoring needs thresholds, owners and response authority. A dashboard without an action rule is observation, not control.
Govern agents by authority and action
Direct answer: Agent governance should control identity, permissions, tool use, memory, approval boundaries, action limits and safe-stop mechanisms.
Agents add risk because they can plan, can hold context, can call tools, can act. Use autonomy as a force multiplier . Writing an email, sending it or updating the customer record that triggered the task all create different exposure for the same model.
Give agents their own scoped identities instead of inheriting a user’s broad access. Allowlist destinations and tools Where possible, limit transaction value, frequency, data volume and action type in the workflow. Provide the reviewer with sufficient source information and context to make a real decision, with human approval before irreversible or high-impact steps.
Control memory and context, and check if untrusted content can redirect goals or manipulate tool calls. OWASP Top 10 for Agentic Applications 2026 lists risks such as goal hijacking, tool misuse, identity and privilege abuse, memory poisoning, and cascading failures. Use those categories to build tests specific to your workload, not as a compliance checkbox list.
Another useful reference for autonomy, human accountability and agent deployment is Singapore’s 2026 Model AI Governance Framework for Agentic AI. It can aid in control design as well as security-oriented frameworks.
Every agent needs a safe way to stop. Decide how to revoke credentials, suspend access to tools, halt queued work, undo supported actions and preserve evidence for investigation. Multi-agent systems require authenticated participants, validation of messages, and restrictions on delegation or expansion of authority.
Govern data across source, use and retention
Direct answer: AI data governance should cover prompts, uploaded files, retrieval sources, embeddings, outputs, evaluation sets, logs, memory and feedback, not just model training data.
For each data class, record source, owner, sensitivity, permitted uses, location, retention, deletion process and downstream derivatives. Confirm rights for development and production separately. A dataset licensed for internal analysis may not permit model training or external processing.
Retrieval systems need source-level controls. The index should preserve access restrictions and respond to revocation. Test freshness, source coverage, conflicting documents and retrieval failure. A citation can show which source was used, but it does not prove the generated claim is supported. Evaluation must test that relationship.
Deletion may need to reach indexes, caches, logs, evaluation sets or fine-tuned artifacts. Test the deletion and revocation path instead of relying on a policy statement.
Govern vendors through shared responsibility
Direct answer: Buying a model, platform or managed service changes who performs controls. It does not remove the enterprise’s accountability for how the system is used.
The decision to build internally, buy a platform or work with a managed provider should establish these responsibilities before supplier selection and deployment.
| Control area | Enterprise responsibility | Vendor evidence |
| Intended use | Approve workload and users | Supported and restricted uses |
| Data handling | Classify data and authorize use | Locations, retention, subprocessors and deletion |
| Model change | Assess impact on the workload | Version notice, release notes and deprecation period |
| Security | Configure identity and monitor use | Independent reports, incident terms and technical controls |
| Evaluation | Test the end-to-end system | Model documentation and relevant provider tests |
| Exit | Maintain continuity and approve migration | Export, transition support and deletion confirmation |
Contract issues include data use, confidentiality, residence, sub-processors, security notification, service levels, audit evidence, material model change, portability and termination. Match due diligence to the system’s impact and dependency upon the supplier.
Don’t take a provider’s generic safety or compliance claim at face value. What product, version, region and control is the evidence for? Also follow concentration risk. A single point of failure across the portfolio can occur when several products share the same foundation model, cloud region or data processor.
Use one lifecycle with risk-based gates
| Lifecycle stage | Governance action | Required evidence |
| Intake and classification | Record purpose, owners, users, data, vendors, actions and worst credible outcomes | Provisional tier and prohibited or disqualifying conditions |
| Design and build | Convert the tier into architecture, evaluation, data and oversight requirements | Control design and planned evidence |
| Validation and release | Test against acceptance thresholds and document residual risk | Versioned evidence and go, conditional go or no-go decision |
| Operation and change | Monitor outcomes, route incidents and trigger reassessment after material change | Logs, thresholds, incidents, change records and reassessment decisions |
| Retirement | Revoke access, archive records, delete required data and terminate dependencies | Retirement record and downstream confirmation |
The AI risk assessment should sit inside the validation and release gate. Low-risk teams need a short, clear route they will follow. High-risk teams need independent review, deeper testing and stronger separation of duties.
Operate exceptions, incidents and material change
An exception should identify the requirement, reason, risk, compensating control, owner, approver and expiry date. It should not become a permanent route around an inconvenient control. Review open exceptions with the same attention given to new approvals.
Connect AI incidents to enterprise response processes. Add AI-specific signals such as unsafe output, unexpected tool use, retrieval leakage, model drift, vendor change and evaluation regression. MITRE ATLAS can help security teams build threat scenarios, which then need to become workload-specific tests, monitoring and response actions.
Material-change rules keep approval current. A full review is not needed for every minor edit. Reassessment is appropriate when a change can alter impact, authority, data exposure, evaluation validity or supplier dependence. Record the decision and evidence even when targeted testing is enough.
Cylix in practice: multi-agent governance and human oversight
The AI Boardroom case study describes a 20-agent enterprise environment with cross-functional orchestration and human-in-the-loop governance. Multi-agent systems like this make the four-object framework concrete because governance cannot stop at the model. Agent identities, action boundaries, shared data, orchestration and supplier dependencies all affect the operating risk.
Make the framework visible in delivery
The framework becomes real when requirements appear in architecture, backlogs, evaluations, release records, monitoring, exceptions and incident response. Teams should be able to see the evidence they need before a gate. Leaders should be able to reconstruct why a system was approved after a change or incident.
Organizations that need help converting the control structure into roles, forums, templates and routines can use AI governance consulting to build the operating capability around it.
| NEXT STEP: Adapt the Governance Framework to Your AI System and map one workload across models, agents, data, vendors and lifecycle gates. |
