What is an AI control plane?

AI tools are proliferating within companies faster than anyone can keep track of. Teams are adopting Cursor, Claude Code, Copilot, ChatGPT, and dozens of internal agents, often independently and often without IT involvement. Most organizations have no reliable way to know which tools are in use, which company data those tools can access, what is being shared with external AI services, or whether any of it is being measured.

The AI control plane is the architecture that solves these problems. It covers four capabilities — connect, control, secure, and observe — across every AI agent and every system those agents reach.

The rest of this guide walks through why this layer is now necessary, how it differs from the AI governance tools enterprises already have in place, and what it looks like in practice.

The problem with AI adoption at scale

Companies want to adopt AI because it makes teams faster and produces better work. The budget is there, the appetite is there, and AI agents like Cursor, Claude Code, Copilot, and ChatGPT have already found their way into daily workflows across engineering, sales, finance, and ops.

The problem is execution. Rolling AI out consistently across an organization is genuinely challenging. Most companies that try run into the same set of gaps:

• No central place to provision MCP servers, skills, and other tools. Every team picks its own.
• No consistent identity layer. People connect personal accounts to enterprise data.
• No visibility into what is being used, by whom, with what data.
• No way to enforce policy on tools IT cannot see.
• No measurement of adoption, and therefore no way to know whether the investment is working.

Locking everything down kills adoption, but leaving everything open invites accumulating incidents. Neither approach holds.

Most enterprises, when they run an audit, discover they have dozens of AI tools in use across the organization, most of which were never approved and none of which are centrally managed. This is the shadow AI problem, and it is already widespread. The cost of doing nothing is clear: data is leaked through consumer AI tools, prompt-injection attacks target agent systems, and enterprises face regulatory pressure from the EU AI Act. The question for most organizations has moved from whether to govern AI to how to govern AI.

AI governance tools today

Several categories of tooling have emerged to address parts of the AI governance problem.

LLM gateways (Portkey, LiteLLM) sit between applications and language models. They handle routing across providers, API key management, rate limiting, caching, and cost tracking. They operate at the model call layer.

MCP gateways and MCP security tools (Speakeasy, Runlayer, MintMCP) sit in front of MCP servers and handle authentication, authorization, and inspection of tool calls. The Model Context Protocol (MCP) is the emerging standard for how AI agents connect to external tools and data.

Identity and access controls (Okta, Azure AD, Google Workspace) provide SSO and enterprise identity infrastructure. Most enterprises already have these in place, though they were not designed for AI agents.

Observability tools (Langfuse, Arize, Galileo) provide visibility into what AI systems are doing. They record which tools are being called, and what outcomes they produce, by which users, and with what frequency.

Policy and threat detection tools (Lakera, Fiddler, Treza) inspect prompts, responses, and tool calls in real time, looking for PII leakage, prompt injection, data exfiltration, and policy violations. The primitive that makes this possible at the AI agent itself is the agent hook, a lifecycle handler that fires on every prompt and tool call. Some entrants, such as Treza, anchor enforcement in hardware attestation and Trusted Execution Environments, giving each agent a cryptographic identity, external policy enforcement, and a tamper-evident audit trail.

What each tool misses

Each category solves a real problem. However, none of them covers the full picture because each sees a different part of the same interaction:

• LLM gateways see the model call, but have no visibility into the user who triggered it, the tool that made the request, or whether the data returned should have been accessible.
• MCP gateways cover the tool call layer, but have no visibility into the model layer above it or the identity layer behind it.
• Identity providers know who the user is, but have no visibility into which AI tools that user is running or what those tools are doing with company data.
• Observability tools record what happened, but cannot enforce policy on it in real time.
• Policy and threat detection tools can inspect and block traffic, but have no way to provision access or manage identity.

No single tool has a correlated view across all these layers. An incident that starts with a user authenticating through SSO, routes through an LLM gateway, triggers an MCP tool call, and leaks data in the response would leave traces in five different systems with no common thread connecting them.

Why incumbent vendors won’t close the gap

Most enterprises already have vendors shipping AI governance features. The question is whether any of their architectures can be extended to cover a problem whose scope exceeds what they were built for.

What AI governance actually requires

A system that governs AI needs to sit between every AI agent and every system those agents can reach. To do that, it needs to:

• Know who the user is and what they are permitted to access
• See every model call, regardless of which provider serves it
• See every tool call, regardless of which MCP server handles it
• Enforce policy on the content of traffic in real time
• Produce a correlated view across all of the above

No existing vendor’s architecture was built to do all of this. Each incumbent occupies a different layer, and the gap between that layer and the oversight the governance problem requires is structural.

Hyperscalers

AWS Bedrock, Azure AI Studio, and Google Vertex are real products with genuine governance capabilities. They provide model routing, access controls, logging, and audit trails at enterprise scale. For an organization running entirely on one cloud, they are a reasonable starting point for AI governance.

A hyperscaler’s controls only apply to traffic routed through its platform, which means:

• An organization running models from Anthropic, Mistral, and an open-source provider gets three separate governance views with no correlation between them.
• Internal agents and MCP-connected tools that do not route through the cloud platform are invisible to these controls.
• As model and tool diversity increases, the share of traffic that the hyperscaler can see shrinks.

Enterprise platform vendors

ServiceNow, CrowdStrike, Salesforce, and the major GRC and ITSM vendors own the policy and risk layer that AI governance needs to connect to. Governance belongs next to identity, policy, and compliance infrastructure, and these vendors already have all of that.

The problem is that these vendors sit above the traffic layer, not inside it:

• They can define policy, but enforcing it on every prompt, response, and tool call would require them to be in the path of that traffic.
• Adding real-time traffic enforcement means building a new layer from scratch, the direction Salesforce’s MuleSoft Agent Fabric and ServiceNow’s AI Control Tower are now moving, each adding an AI gateway for MCP governance, with ServiceNow’s AI Control Tower extending to real-time enforcement that can shut down a rogue agent.

Point tools

LLM gateways and MCP security tools are already inside the traffic. They have genuine enforcement capability within, and deep technical knowledge of, their respective layers.

Expanding their capabilities beyond their layers requires building what they do not have:

• Visibility into the identity layer above them and the policy layer that needs to govern them
• Cross-layer correlation across model calls, tool calls, and user identity

Most point tools in infrastructure categories are acquired before they can complete that expansion.

Each of these vendors was built to own one layer. AI governance requires owning the path between layers, and none of them is positioned to do that without rebuilding significant parts of their architecture.

What is the AI control plane?

The term “control plane” originated in networking, where Kubernetes, service meshes, and Cloudflare already have control planes for the same reason: they name the part of the system that governs the rest of the system.

Enablement and governance

The AI control plane has two jobs. The first is enablement: rolling out AI capabilities across the organization so every team can use them. The second is governance: making sure that rollout doesn’t cause a security incident, a data leak, a compliance violation, or a regulatory problem.

These two jobs pull in opposite directions. Enablement wants more tools, more data connections, and broader access. Governance wants every interaction inspected, every permission scoped, and every action audited.

Splitting them across two teams always fails

The instinctive organizational response is to assign enablement and governance to different teams. For example, IT or security could own governance, and a platform team or newly-hired Chief AI Officer could own enablement.

This split has a consistent failure mode. The governance team, lacking visibility into what people actually need from AI, designs controls that block the most useful capabilities. The enablement team, under pressure to ship, routes around the controls or negotiates exceptions. The result is either a governance posture that exists on paper while employees use personal ChatGPT accounts for real work, or an adoption program that moves fast and creates incidents the security team discovers in postmortems.

A single layer resolves the conflict

This failure is structural. Governance and enablement look like opposing forces because organizations treat them as separate layers. When the same layer that routes AI traffic also enforces policy on it, governance and enablement are no longer in tension.

Consider the networking analogy: Kubernetes does not have a separate team for allowing workloads to run and another for ensuring workloads run safely. The control plane does both, by design.

The four functions

An AI control plane should do four things:

Connect

It should bring every AI agent (Claude, ChatGPT, Cursor, Copilot, Codex, internal agents, and product agents) and every system that matters (SaaS tools, internal APIs, databases, and skills) onto a single plane, with per-team registries and SSO-integrated identity.

The result is that new AI capabilities reach the right teams in days instead of months, and IT stops chasing a moving target of ad-hoc integrations.

Control

It should enforce who can use what and under what conditions by scoping access to teams or roles, implementing credential management that keeps API keys out of config files, using OAuth 2.1 where the protocols support it, and maintaining a full audit trail.

The shift that matters here is that policies become versioned, testable, executable rules applied automatically at the point of use. The control plane evaluates policy on every request, unlike a wiki document that relies on people remembering to follow it.

The policy on paper becomes the policy at runtime, and audits stop surfacing questions the CISO can’t answer.

Secure

It should inspect every prompt, response, and tool call in real time, with active blocking of PII and data exfiltration, passive detection of prompt injection and shadow MCPs, and integration with existing SIEM and security tooling rather than replacing it.

Incidents become detectable in real time, and the response team works from the tooling they already know.

Observe

It should measure what’s actually happening. By recording token use by team, agent, tool, and user, then tracking that use against organizational targets, the control plane produces data that proves whether the AI investment is working.

Leadership gets real numbers behind the AI investment rather than anecdotes, and can distinguish genuine adoption from theatre.

AI guardrails and AI risk management

Two terms show up constantly in vendor pitches and analyst reports: AI guardrails and AI risk management. The AI control plane is how an enterprise operationalizes both.

AI guardrails are the runtime controls that constrain what AI agents are allowed to say and do, blocking PII leakage, prompt injection, unsafe tool calls, and data exfiltration. In the control plane, this is the Secure function: real-time inspection of every prompt, response, and tool call, integrated into existing SIEM tooling rather than replacing it. The threat models these guardrails address are covered in detail in AI security.

AI risk management is the broader program of identifying, measuring, and mitigating the risks AI introduces, whether regulatory, security, operational, or reputational. The control plane is the execution layer for that program. Connect gives risk teams a real-time inventory of every AI agent and tool in use. Control turns policy on paper into rules enforced at runtime. Secure detects active threats. Observe produces the audit trail and metrics that risk and compliance teams need to evidence the program.

These are not competing categories. They are different layers of the same architecture, and the control plane is what lets one organization run both.

AI control plane vs AI governance framework

It is easy to conflate the two, but they sit at different levels. An AI governance framework describes what an organization should do; an AI control plane is what actually does it.

Most AI governance programs are built on published frameworks: NIST AI RMF, ISO/IEC 42001, the EU AI Act, and the internal policy decks that follow them. They tell an organization to identify AI use cases, classify risk, document controls, monitor outcomes, and review periodically. The framework is the program. For how the major frameworks compare and where each one stops, see AI security frameworks.

An AI control plane is the runtime that executes the program. The framework says “log every interaction with sensitive data”; the control plane is what intercepts the interaction and writes the log. The framework says “scope access by role”; the control plane is what evaluates the role on every request and decides what passes.

The two are complementary. A governance framework without a control plane is a document. A control plane without a framework is infrastructure with nothing telling it what to enforce. Tools that ship policy templates without runtime enforcement leave the gap between policy and behavior open, and the control plane is what closes it.

What a mature AI control plane enables

When the pieces are in place, the operational shape of the organization changes in a handful of specific ways.

AI reaches every team quickly

New tools get added to the central registry once and are available to the teams entitled to use them. Teams stop standing up their own integrations in parallel, and new AI capabilities reach the right people in days rather than months.

Consistent permissions across every tool and agent

An engineer using Cursor, a salesperson using ChatGPT Enterprise, and an analyst using an internal agent are all subject to the same permission model, because the permissions live in the control plane rather than in each agent.

Security visibility into the AI layer

Security has visibility into AI in the same way it has visibility into the network, the endpoints, and the cloud, enabling real-time incident detection and providing data for audits to draw from.

Adoption metrics for leadership

The AI control plane gives leadership actual numbers for tracking adoption by team, discerning which tools are producing outcomes and which are shelfware, and seeing where AI is moving the business. Leaders can base investment decisions on concrete metrics rather than anecdotes.

Reusable infrastructure for every subsequent project

New AI initiatives ship on shared infrastructure instead of rebuilding the connection, identity, and governance layers every time. The second and tenth AI projects cost a fraction of the first.

The components for doing each of these things exist today. What the control plane provides is the architectural foundation that makes them work together.

Things to consider before deploying a control plane

These costs are real enough to shape whether a deployment succeeds.

Latency

Adding the control plane to every model call and tool call means adding a network hop. For interactive use cases, the overhead is typically small, but for agentic workflows that chain many calls in sequence, it compounds. The content inspection layer adds further delay if it runs synchronously.

• Agentic loops with 20 or more sequential calls can accumulate several seconds of added latency.
• Synchronous PII inspection on every response adds delay on the critical path.
• Asynchronous inspection reduces latency but means some traffic is not caught in real time.

Vendor lock-in

The AI control plane category is less than two years old. The vendors building in this space are early, and the architecture has not yet settled.

• A vendor’s architecture may shift significantly as the category matures.
• Acquisition can change product direction quickly in early-category markets.
• Deep integration with a single vendor’s identity and policy model makes migration expensive.

Mitigating lock-in means preferring open standards at the integration points. Use components that you can swap without rebuilding everything connected to them, such as OAuth 2.1 for identity, OpenTelemetry for observability, and MCP for tool connections.

A note on Speakeasy

Speakeasy is building the AI control plane. We started with the connection and identity layer, the first place companies typically get stuck when moving beyond ad-hoc AI adoption, and have been extending across the four functions since. The coverage map below shows where we are today.

The reference architecture in this guide reflects what a complete solution needs to look like. It is the architecture we are building toward. If you are an engineering or platform team figuring out how AI should flow through your organization, get in touch.