All Azure Technologies @ one Place

10 Decisions That Make or Break Your Enterprise AI Platform

5/3/2026

Every AI platform discussion starts with a diagram and ends with a disagreement. Not about boxes and arrows, but about choices: which tool teams build with, where prompts are processed, how much an agent may do on its own, and who pays for the tokens. Here are the 10 decisions I push to settle early, with what I usually pick and what to watch out for.

Why Decisions Beat Diagrams

Every AI platform discussion I join starts with a diagram and ends with a disagreement. Not about boxes and arrows, but about choices: which tool teams should build with, who pays for tokens, how much an agent may do on its own.

Those choices get made whether you make them or not. Leave them open and every team makes its own, and six months later you have ten different answers and no platform.

These are the 10 decisions I push to settle early, with the options, what I usually pick, and what to watch out for.

TipTurn each decision into a one-page Architecture Decision Record (ADR): the question, your choice, why, and who can approve an exception. Agreement with my picks is optional; writing yours down isn't.

At a Glance

#	Decision	What I pick
01	Who builds what	Pick per use case
02	Capacity model	Start pay-as-you-go, graduate to PTU
03	Where prompts are processed	Classify data, then choose
04	How the model knows your business	RAG first
05	Kill the API keys	Microsoft Entra ID everywhere
06	Network isolation	Bring your own VNet
07	Where guardrails live	Layered by default
08	How much autonomy agents get	Autonomy earned per action
09	Evaluations as a release gate	Automated evals in CI/CD
10	Who pays for tokens	Showback, then chargeback

The 10 Decisions

Decision 01

Who builds what

The question: Copilot Studio, Microsoft Foundry, or fully code-first? Most organizations end up with all three, and fight about it.

CHOICESOne tool for everythingPick per use case

What I pick: Pick per use case

Copilot Studio for business-led agents that live in Teams and Microsoft 365, built by makers with low-code.
Foundry for pro-dev agents that need custom models, tools, evaluations and private networking.
Code-first (Agent Framework, Semantic Kernel, LangGraph on Container Apps or AKS) when you need full control of orchestration.
Publish a one-page decision guide so teams stop debating it per project.

Watch out: shadow agents. Whatever the tool, every agent needs an owner, an identity and an entry in an inventory.

Decision 02

Capacity model

The question: Pay-per-token, provisioned throughput (PTU), or batch?

CHOICESPTU from day oneStart pay-as-you-go, graduate to PTUBatch for everything

What I pick: Start pay-as-you-go, graduate to PTU

Start on Standard (pay-per-token) while usage is unpredictable.
Move steady, latency-sensitive production traffic to PTU once you can size it, and use spillover to send bursts back to Standard instead of returning 429s.
Push offline jobs (document enrichment, bulk summarization) to Batch for roughly half the price.

Watch out: PTU is a commitment. Size it from real telemetry, not from a slide.

Decision 03

Where prompts are processed

The question: Global, Data Zone or regional deployments? This is a compliance decision disguised as a pricing option.

CHOICESGlobal by defaultClassify data, then choose

What I pick: Classify data, then choose

Global deployments can process prompts in any Azure region; they're cheapest and get models first.
Data Zone deployments keep processing inside the US, EU or APAC zone; Standard and Regional Provisioned stay in your geography.
Data at rest stays in the resource's geography for every type. Map your data classes to deployment types once, and enforce it with policy.

Watch out: the fine print of tools and features (evaluators, voice, web grounding). They can process data somewhere else even when your model doesn't.

Decision 04

How the model knows your business

The question: RAG, fine-tuning, or just stuffing everything into a long context window?

CHOICESFine-tune firstRAG firstLong context only

What I pick: RAG first

RAG on Azure AI Search keeps answers grounded, citable and up to date without retraining.
Respect document permissions at retrieval time (security trimming), or your agent becomes a very polite data leak.
Fine-tune later, for tone, format or narrow tasks, once RAG is measured and stable.

Watch out: chunking and index design. Most "the model is dumb" complaints are really retrieval problems.

Decision 05

Kill the API keys

The question: How do apps and agents authenticate to models and data?

CHOICESAPI keys in Key VaultMicrosoft Entra ID everywhere

What I pick: Microsoft Entra ID everywhere

Managed identities for apps, and Microsoft Entra Agent ID so agents get their own governable identities.
On-behalf-of flows when an agent must only see what the user is allowed to see.
Turn off key-based auth on AI resources (disableLocalAuth) and enforce it with Azure Policy.

Watch out: keys hidden in notebooks, CI variables and demo repos. Scan for them before you flip the switch.

Decision 06

Network isolation

The question: Managed virtual network, your own VNet, or public endpoints with IP rules?

CHOICESPublic + IP rulesFoundry managed VNetBring your own VNet

What I pick: Bring your own VNet

If you have an enterprise hub (firewall, DNS, inspection), bring your own VNet and plug into it.
Plan subnets early: the agent subnet is delegated to Microsoft.App/environments, can't be shared, and a /24 is recommended.
Link the private DNS zones (privatelink.cognitiveservices.azure.com, privatelink.openai.azure.com, privatelink.services.ai.azure.com plus Search, Cosmos DB and Blob) before the first deployment.

Watch out: if you have no hub yet, Foundry's managed VNet with allow only approved outbound is a fast secure start, but you can't switch the mode back later.

Decision 07

Where guardrails live

The question: Built-in model filters, a gateway layer, application code, or all of them?

CHOICESEverything in app codeLayered by default

What I pick: Layered by default

Use the model's built-in content filters and Prompt Shields (including document attacks for RAG) as the baseline.
Add a gateway-level check for models that don't ship with Azure's filters (open-source or third-party).
Keep business rules (what a user may trigger) in the application, where the context is.

Watch out: stacking three safety layers on every call. Measure the latency and false-positive rate before you add another.

Decision 08

How much autonomy agents get

The question: Suggest, act with approval, or act on their own?

CHOICESFully autonomousAutonomy earned per action

What I pick: Autonomy earned per action

Read-only tools can run freely; anything that writes, sends or pays gets a human approval step first.
Give each agent least-privilege tool permissions, not the developer's permissions.
Raise autonomy per action once evaluations and logs show it behaves.

Watch out: tool sprawl. An agent with 40 tools is harder to secure and usually worse at choosing the right one.

Decision 09

Evaluations as a release gate

The question: How do you know a new prompt, model version or tool didn't make things worse?

CHOICESManual spot checksAutomated evals in CI/CD

What I pick: Automated evals in CI/CD

Keep an evaluation dataset in Git: happy paths, edge cases, known failures and attack prompts.
Run it on every change with the Foundry evaluation GitHub Action (or Azure DevOps) and block the release on regressions.
Keep evaluating in production, because user behavior drifts even when your code doesn't.

Watch out: vanity metrics. Groundedness and task success matter more than a high average "quality" score.

Decision 10

Who pays for tokens

The question: Is AI a central cost, or does every product team own its spend?

CHOICESCentral budgetShowback, then chargeback

What I pick: Showback, then chargeback

Meter tokens per app and team from day one, even if nobody is billed yet.
Start with showback so teams see their cost; move to chargeback once numbers are trusted.
Set budgets and alerts per team. Runaway agent loops are a real incident type now.

Watch out: a free central budget with no visibility. It feels generous until the first invoice review.

What Ties Them Together

1Owners, not committees

Each decision needs one accountable owner and a simple exception path. Committees are where decisions go to wait.

2Check the region first

Model availability, deployment types and agent features differ per region. Confirm what's available where you deploy before you commit, especially for decisions 2, 3 and 6.

3Put an expiry date on every ADR

The AI stack changes every quarter. Review these decisions every six months, or they'll quietly go stale.

My Take

The best AI platforms I've seen aren't the ones with the most services. They're the ones where every team knows the answers to these ten questions before writing a line of code. Decide once, write it down, and let teams move fast inside the lines.

Previously in this series: Your LLMs Need a Front Door: Why Every Enterprise AI Platform Needs an AI Gateway. Next up: AI networking on Azure in depth.