Azure API Management Adds Multi-Model AI Control Plane

Microsoft put Azure API Management's AI Gateway tier into public preview on July 27, available in East US 2 and Sweden Central at no cost while pricing is determined. The tier is a purpose-built resource whose control plane is organized around models, MCP servers, and tools rather than APIs — a structural departure from the policy-layering approach of the classic and v2 tiers, which retain their existing AI gateway capabilities unchanged.

The model federation layer covers the multi-cloud reality most enterprise fleets already live in. The preview publishes Foundry-hosted models including OpenAI, Anthropic, and Mistral alongside models in AWS Bedrock, Google Vertex AI, and OpenAI direct. All OpenAI-compatible providers share one endpoint path; the gateway routes on an exact match of the model field, so each published model requires a unique name. Anthropic runs through a custom provider with Messages API passthrough. A gateway provisions in about a minute with no scale units to plan.

Tool federation extends the same pattern to the MCP layer. Teams can expose an existing MCP server over SSE or Streamable HTTP, convert REST API operations into an MCP server by uploading an OpenAPI spec, or draw on more than 1,400 connector-backed tools from the Power Platform and Logic Apps library without hosting a server. Multiple MCP servers federate behind a single endpoint, so an agent connects once and resolves tools across all of them. Per-backend authentication supports API keys, OAuth 2.0 client credentials, managed identity, and mTLS.

Governance is configured through policy cards in the portal as JSON properties rather than the XML expressions APIM veterans know. Cards cover request and token rate limits, token quotas, Azure AI Content Safety, and fallback to a secondary model. Policies apply per asset, making coverage for each model or MCP server explicit. Telemetry flows as OpenTelemetry token metrics with GenAI semantic conventions to Application Insights, Datadog, Splunk, Grafana Cloud, or any OTLP endpoint the customer controls. The resource runs in the customer's own subscription and Entra tenant.

The operating model splits platform ownership from team autonomy. A central group connects approved models and tools, sets guardrails, and retains the usage picture; application teams test assets in a built-in console and build against them without routing every change through the center. This works only if the access boundary holds. It does not hold the way APIM subscription scoping did: a runtime key is gateway-scoped, reaching every model and every tool published on that gateway. Microsoft's guidance is one key per application, but a leaked key's blast radius is the entire gateway rather than a single product. Teams that relied on APIM subscriptions to scope consumers to specific APIs must redesign that boundary.

Two unresolved questions from architects matter. Agent lifecycle coverage is the first: if an agent run ends without clean completion after producing useful work, it is unclear whether the output is preserved for auditable review or whether the gateway retries the run from scratch. The distinction matters for stateful agent pipelines where idempotency is not guaranteed. The second is coexistence: organizations that built AI gateway configurations on Premium or Standard v2 have no published guidance on whether those investments carry across, run alongside, or migrate into the new tier. Microsoft's documentation describes the AI Gateway tier as an extension of the existing gateway, but the portal and control plane are structurally separate.

Preview posture requires careful reading before planning an onboarding schedule. There is no SLA. APIs, telemetry, limits, regions, and pricing can all change before GA. Specific throughput and quota caps are unpublished. Pricing is not yet announced, which leaves the cost-governance argument—the one most cited as the tier's practical value—as the least settled part of the release.

The architectural takeaway: the AI Gateway tier solves the control-plane consolidation problem for teams fronting five or more model providers, but key-scoping and coexistence gaps make it a workbench deployment rather than a production control plane until Microsoft ships quota granularity and migration guidance.