Most teams meet Azure AI Foundry the same way they meet every new AI platform: open the portal, deploy a model, paste an endpoint into a notebook, and celebrate the first completion. That path is fine for learning. It is a terrible foundation for anything that will touch customer data, cost money at scale, or survive an audit.

This post is the path I wish more teams took. We start with the smallest useful call to a model, then layer on the pieces that turn a demo into a platform: policies, governance, infrastructure as code, GitHub repository layout, operational best practices, and a real-world architecture you can actually implement.

The goal is not to memorize every Foundry blade. The goal is to understand the control plane around models so you can ship AI features without handing the business an ungoverned API key and a prayer.

What Azure AI Foundry Actually Is

Azure AI Foundry (the evolution of Azure AI Studio / the Microsoft Foundry experience) is not just a model catalog. Treat it as an enterprise control plane for AI workloads:

  • Model catalog and deployments — choose, deploy, and version foundation models and custom fine-tunes.
  • Project and hub topology — shared platform resources with team-scoped projects on top.
  • Connections — governed links to data, search, storage, Key Vault, and other Azure services.
  • Safety and evaluation — content filters, evaluation loops, and responsible AI controls.
  • Identity and networking — managed identities, private endpoints, and VNet-integrated compute.
  • Observability — usage, latency, failures, and audit signals that feed Azure Monitor.

If you only use Foundry as “the place I click Deploy model,” you will rebuild half of a platform ad hoc in application code. Use it as the platform layer, and applications become thinner and safer.

Mental Model: Hub, Project, Deployment, Consumer

Keep four objects clear in your head:

  1. Hub (or shared AI resource boundary) — platform-owned shared capacity, networking, identity, and baseline connections.
  2. Project — team or product-scoped workspace that inherits hub controls.
  3. Deployment — a specific model endpoint with SKU, version, rate limits, and content filters.
  4. Consumer — your app, agent, batch job, or APIM product that calls the deployment with a workload identity.
Platform team owns hub + policies + networking
  → Product teams own projects + apps
    → Apps call deployments through a controlled gateway
      → Every call is authenticated, logged, and attributable

That separation is the difference between “we have AI somewhere in Azure” and “we run AI as a product.”

Zero: Prerequisites Before the First Call

Before you chase completions, lock down the boring foundations:

  • An Azure subscription with a clear owner and cost center.
  • Entra ID groups for ai-platform-admins, ai-project-contributors, and ai-app-readers.
  • A resource group strategy: one shared platform RG, separate RGs per environment (dev, test, prod).
  • Decision on public vs private networking. Private networking is painful to retrofit; decide early.
  • A Key Vault for secrets you cannot eliminate yet — and a plan to eliminate most of them with managed identity.
  • Log Analytics workspace and a naming convention that includes environment, region, and workload.

Do not skip the identity model. Static API keys in app settings are how demos become incidents.

One: Call Your First Model the Right Way

Deploy a model

In Foundry, create (or reuse) a project, open the model catalog, and deploy a capable chat model appropriate for your region and compliance needs. Capture:

  • Endpoint URL
  • Deployment name (not just the model family name)
  • API version you intend to pin
  • The identity or key you will use from a non-portal client

Prefer Microsoft Entra ID authentication with a managed identity or app registration over key-based auth as soon as the SDK path supports your scenario.

Minimal Python call (Entra ID)

import os
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import OpenAI

endpoint = os.environ["AZURE_AI_ENDPOINT"]  # Foundry / Azure OpenAI endpoint
deployment = os.environ["AZURE_AI_DEPLOYMENT"]  # deployment name

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(),
    "https://cognitiveservices.azure.com/.default",
)

client = OpenAI(
    base_url=f"{endpoint.rstrip('/')}/openai/v1/",
    api_key=token_provider,
)

response = client.chat.completions.create(
    model=deployment,
    messages=[
        {"role": "system", "content": "You are a concise platform engineering assistant."},
        {"role": "user", "content": "Explain hub vs project in one paragraph."},
    ],
    temperature=0.2,
    max_tokens=300,
)

print(response.choices[0].message.content)

What “right” means on day one

Even the first script should demonstrate production habits:

  • Pin the deployment name in config, not hard-coded model marketing names.
  • Use environment variables or App Configuration, never secrets in source.
  • Log request IDs / correlation IDs from responses when available.
  • Set modest max_tokens and temperature defaults; creative chaos is not a platform default.
  • Fail closed on auth errors instead of falling back to a shared key “just for now.”

Curl-shaped mental check

If you can explain the call as “authenticated principal → endpoint → deployment → completion with content filter,” you understand enough to move on. If your mental model is still “paste key into Postman,” pause and fix identity first.

Policies: Make the Guardrails Explicit

Policies are how you stop every team from inventing a slightly different unsafe pattern.

Azure Policy for model deployment control

Use Azure Policy at management group or subscription scope to constrain what can be deployed:

  • Allow only approved model publishers and model families.
  • Restrict regions to those approved for data residency.
  • Require specific SKUs or deny oversized deployments in non-prod.
  • Deny public network access where private endpoints are mandatory.
  • Require diagnostic settings on AI accounts and related resources.
  • Require tags: owner, costCenter, dataClassification, environment.

Built-in and custom policy definitions around Foundry / Azure AI model deployment are the difference between a catalog and a free-for-all.

Content safety and application policies

Platform policy is not only Azure Policy:

  • Content filters on deployments for hate, self-harm, sexual, and violence categories with environment-specific thresholds.
  • Groundedness / protected material controls where your scenario needs them.
  • API Management policies for token budgets, IP restrictions, JWT validation, and abuse detection.
  • Prompt and tool policies in your orchestration layer: which tools an agent may call, which systems of record are writable, and what requires human approval.

Write these as code and config, not wiki pages. A policy that only exists in a Confluence page is a suggestion.

Example policy intent (human-readable)

Non-production subscriptions:
  - may deploy approved chat and embedding models only
  - must use Entra ID auth for applications
  - may not disable content filters
  - must emit diagnostics to the central Log Analytics workspace

Production subscriptions:
  - everything above, plus
  - private endpoints required
  - customer-managed keys where mandated
  - no direct keys distributed to developers
  - all app traffic enters through APIM

Governance: Who Owns What, and How Change Happens

Governance fails when everyone can deploy anything and nobody owns the blast radius.

RACI that actually works

Concern Platform team Product team Security / Compliance FinOps
Hub, networking, shared connections A/R C C C
Project creation standards A R C I
Model allowlist and policies A C A/R C
Application prompts and tools C A/R C I
Cost budgets and alerts C R I A/R
Incident response for AI endpoints A/R R C I

A = accountable, R = responsible, C = consulted, I = informed.

Lifecycle governance

Govern the full lifecycle, not just deploy day:

  1. Request — product team requests a model or project with data classification and use case.
  2. Approve — security and platform review against allowlist, residency, and risk tier.
  3. Provision — IaC pipeline creates project, deployment, identities, and diagnostics.
  4. Operate — SLOs, budgets, evals, and access reviews run continuously.
  5. Change — model version upgrades go through the same pipeline with eval gates.
  6. Retire — decommission deployments, revoke identities, archive logs and eval evidence.

Data and risk tiers

Classify AI use cases the same way you classify data:

  • Tier 0 — public / low sensitivity assistants with no enterprise data.
  • Tier 1 — internal knowledge with standard corporate data.
  • Tier 2 — customer data, regulated content, or high-impact automated actions.
  • Tier 3 — safety-critical or legally constrained automation; human-in-the-loop mandatory.

Higher tiers demand stricter networking, stronger evaluation, tighter RBAC, and more complete audit evidence.

Agent and application inventory

If you ship agents, maintain an inventory: owner, environment, model deployment, tools, data sources, risk tier, and last access review. An untracked agent is an untracked production service with a stochastic control path.

Infrastructure as Code: Foundry as a Platform Product

Click-ops creates snowflake AI environments. IaC creates a product.

What to put in code

  • Resource groups, hub/project topology, and model deployments
  • Role assignments for groups and managed identities
  • Private endpoints, DNS, and network security group baselines
  • Key Vault, App Configuration, and connection objects
  • Diagnostic settings and metric alerts
  • APIM APIs, products, subscriptions, and policies
  • Azure Policy assignments and exemption process (exemptions as code too)

Bicep module sketch

param environmentName string
param location string
param projectName string
param modelName string
param deploymentName string
param capacity int
param logAnalyticsWorkspaceId string

// 1) AI account / Foundry-aligned resources
// 2) Project-scoped RBAC
// 3) Model deployment with required content filters
// 4) Managed identity for consuming app
// 5) Diagnostic settings
// 6) Optional private endpoint

output endpoint string
output deploymentName string
output managedIdentityClientId string

Keep modules opinionated. A good platform module should make the secure path the default path: diagnostics on, public access off in prod parameters, required tags enforced, approved model parameter validated against an allowlist.

Terraform is fine too

If your org standard is Terraform, the same boundaries apply. The tool matters less than:

  • remote state with locking
  • separate plan/apply identities
  • environment promotion through pipelines
  • policy-as-code checks before apply

Environment promotion

dev  → automated apply on merge to main (or develop)
test → apply on release candidate tag
prod → apply on approved release with manual protection rule

Never “hot fix in portal prod” and promise to codify it later. Later rarely comes.

GitHub Repos: Structure That Scales Beyond One Demo

A single monorepo can work. Multiple repos can work. Chaos repos do not.

  1. ai-platform-infra — Bicep/Terraform for hubs, shared networking, policy assignments, APIM baselines.
  2. ai-project-templates — cookiecutter or Template Repository for new product projects (app skeleton, identity wiring, eval harness, CODEOWNERS).
  3. ai-policy-as-code — Azure Policy definitions/assignments, APIM policy fragments, OPA/Scorecards if you use them.
  4. product-<name>-ai — product application code, prompts, tool definitions, evaluation datasets.
  5. ai-runbooks (optional) — incident playbooks, model upgrade checklists, exception process.

Inside a product AI repo

/src                  application and orchestration code
/infra                app-level IaC (identity, app service/container, role assignments)
/prompts              versioned prompt assets
/evals                offline eval sets and scorers
/policies             app-level guardrails and tool allowlists
/.github/workflows    ci, eval gates, infra plan/apply, release
/docs                 ADRs and threat model notes

GitHub practices that matter

  • CODEOWNERS for /infra, /policies, and /prompts.
  • Branch protection with required checks: build, unit tests, eval smoke, IaC plan.
  • Environment protection rules for prod with required reviewers from platform + security.
  • OIDC federation to Azure — no long-lived deployment secrets in GitHub if you can avoid them.
  • Signed commits / controlled release tags for production model and prompt changes.
  • PR templates that force the author to state data classification, model used, and eval results.

Pipeline shape

PR opened
  → build + unit tests
  → prompt/policy lint
  → offline eval smoke against fixed dataset
  → infra plan (non-apply)
  → human review

Merge / release
  → infra apply (env-scoped)
  → deploy app
  → post-deploy eval canary
  → budget and latency alert verification

If a model upgrade fails eval gates, it does not ship. “It looked better in chat” is not a release criterion.

Best Practices Checklist

Identity and access

  • Prefer managed identities end to end.
  • Scope RBAC to project and resource, not subscription-wide Contributor for developers.
  • Separate human access from workload access.
  • Run periodic access reviews on AI resource roles.

Networking and data

  • Decide private networking before the first production deployment.
  • Keep training/fine-tune data and logging pipelines inside the approved boundary.
  • Minimize prompt/log retention of sensitive content; redact where possible.
  • Explicitly document egress: which external model calls, if any, are allowed.

Reliability and cost

  • Pin deployment names and API versions; upgrade deliberately.
  • Set token and request budgets per product and environment.
  • Use caching and smaller models for classification/routing where quality allows.
  • Define SLOs: availability, latency (p95), error rate, and eval regression thresholds.
  • Autoscale carefully — capacity planning for model deployments is part of SRE now.

Quality and safety

  • Maintain offline evaluation sets per critical use case.
  • Track groundedness/faithfulness for RAG systems.
  • Require content filters appropriate to the risk tier.
  • Add human approval for high-impact tools and write actions.
  • Version prompts like code; never edit production prompts only in a UI.

Operations

  • Centralize diagnostics in Log Analytics.
  • Alert on auth failures, 429s, latency spikes, cost anomalies, and filter trigger rates.
  • Practice model rollback the same way you practice app rollback.
  • Keep an AI incident runbook: how to disable a deployment, rotate identities, and communicate impact.

Real-World Architecture: Contoso Support Copilot

Here is a concrete target architecture for an internal customer-support copilot that answers from approved knowledge and can draft (but not send) CRM responses.

Scenario

  • Users — Contoso support agents in Entra ID.
  • Data — product docs, policy PDFs, and a subset of CRM case summaries (Tier 2).
  • Actions — retrieve knowledge, summarize a case, draft a reply; sending the reply requires a human click in the existing CRM UI.
  • Constraints — private networking in production, full audit trail, monthly access reviews, cost cap per environment.

Architecture diagram

                            +---------------------------+
                            |  Entra ID + Conditional    |
                            |  Access + PIM (admins)    |
                            +-------------+-------------+
                                          |
                                          v
+------------------+          +---------------------------+
| Support Agent UI |          | Azure API Management      |
| (web app)        |--------->| - JWT validation          |
+------------------+          | - quota / token budgets   |
                              | - correlation IDs         |
                              | - abuse policies          |
                              +-------------+-------------+
                                            |
                    +-----------------------+-----------------------+
                    |                                               |
                    v                                               v
        +-----------------------+                     +-------------------------+
        | Copilot Orchestrator  |                     | Audit / Diagnostics     |
        | - prompt versioning   |                     | Log Analytics / App Ins.|
        | - tool allowlist      |                     | Cost alerts + workbooks |
        | - human approval gate |                     +-------------------------+
        +-----------+-----------+
                    |
      +-------------+--------------+------------------+
      |                            |                  |
      v                            v                  v
+-------------+            +---------------+   +----------------+
| Foundry     |            | Azure AI      |   | CRM connector  |
| deployment  |            | Search (RAG)  |   | read-only      |
| chat model  |            | + indexes     |   | draft-only out |
| + filters   |            +-------+-------+   +----------------+
+------+------+                    |
       |                           v
       |               +-----------------------+
       |               | Knowledge ingestion   |
       |               | docs → chunk → embed  |
       |               | ACL metadata preserved|
       |               +-----------------------+
       v
+------------------+     +--------------------+
| Managed Identity |     | Key Vault          |
| workload auth    |     | secrets (minimal)  |
+------------------+     +--------------------+

Platform plane (separate from product app):
  Hub / shared Foundry resources
  Azure Policy assignments
  Private endpoints + private DNS
  GitHub OIDC deploy identities
  Model allowlist + eval evidence store

Component responsibilities

  • APIM is the only public-ish front door the UI talks to. It enforces authn/authz, quotas, and basic threat protection.
  • Orchestrator owns prompts, tool policy, RAG context assembly, and refusal behavior. It never uses a shared user key to call the model.
  • Foundry deployment serves the chat model with content filters and diagnostics enabled.
  • AI Search stores chunks with ACL metadata so retrieval respects document permissions.
  • CRM connector is read-only for case context and write-disabled for send operations in v1.
  • Platform plane owns hub networking, policy, budgets, and the promotion pipeline.

Implementation steps

Phase 0 — Platform foundation (week 1)

  1. Create management group policy assignments for region, tags, diagnostics, and model allowlist.
  2. Stand up ai-platform-infra with hub resources, Log Analytics, Key Vault, and APIM baseline in dev.
  3. Configure GitHub OIDC to Azure for platform and product deploy identities.
  4. Define Entra groups and PIM for platform admins.

Phase 1 — Project and model (week 1–2)

  1. Provision a Foundry project for support-copilot-dev via IaC.
  2. Deploy approved chat and embedding models with content filters on.
  3. Create the orchestrator managed identity and grant it only the roles it needs to call the deployment and search index.
  4. Prove a single authenticated end-to-end completion through APIM with correlation IDs visible in logs.

Phase 2 — Knowledge path (week 2–3)

  1. Build ingestion for product docs and policies with chunking and metadata (source, acl, version, asOf).
  2. Index into Azure AI Search with permission filters.
  3. Add offline eval set: 50–100 real support questions with expected citations.
  4. Gate PRs on eval smoke (citation presence, refusal on out-of-scope, no critical safety misses).

Phase 3 — Product UI and draft workflow (week 3–4)

  1. Ship the support agent UI behind Entra ID.
  2. Implement tools: search_knowledge, fetch_case_summary, draft_reply.
  3. Enforce tool allowlist in code; no generic HTTP tool in v1.
  4. Store prompt versions in git; include prompt version in every log event.

Phase 4 — Production hardening (week 4–5)

  1. Promote infra to test then prod with private endpoints and private DNS.
  2. Turn on stricter APIM policies, budgets, and alerts (latency, 429, cost, filter rate).
  3. Run access review and threat model review; record exceptions as code.
  4. Execute a game day: disable deployment, rotate identity, fail over to previous model version, verify audit trail.

Phase 5 — Operate and improve (ongoing)

  1. Weekly eval regression on a living dataset.
  2. Monthly model/provider review against the allowlist and cost reports.
  3. Quarterly access reviews and policy exemption cleanup.
  4. Only then consider higher-risk tools (for example, sending CRM replies) with human approval and stronger change control.

Success metrics

  • p95 latency under agreed SLO for draft generation.
  • Groundedness above target on the eval set.
  • Zero unauthenticated model calls in production logs.
  • 100% of prod changes via pipeline (infra, prompt, model version).
  • Cost per solved ticket assist tracked and budget-alarmed.
  • Mean time to disable a bad deployment measured in minutes, not hours.

Common Mistakes

  • Treating Foundry as a notebook endpoint factory instead of a platform.
  • Distributing API keys to every developer laptop and calling it “agile.”
  • Skipping Azure Policy until “we have something working.”
  • Deploying models in portal prod with no IaC representation.
  • No eval set, so model upgrades are vibes-driven.
  • Letting agents hold broad write credentials to enterprise systems.
  • Ignoring 429s and capacity until launch week.
  • Logging raw prompts that contain customer secrets with infinite retention.
  • One shared deployment for every product with no chargeback story.
  • No owner for the hub — platform orphans become shadow IT quickly.

From Zero to Ninja: The Progression

Zero — authenticated first call to a named deployment.
Padawan — project + content filters + diagnostics + config not in source.
Operator — IaC, APIM front door, budgets, alerts, CODEOWNERS.
Guardian — Azure Policy, private networking, eval gates, access reviews.
Ninja — productized platform: templates, automated promotion, inventory of apps/agents, deliberate model lifecycle, and a real architecture that survives audits and incidents.

Final Perspective

Azure AI Foundry rewards teams that treat AI like any other serious enterprise runtime: identity first, policy as code, boring promotion paths, and clear ownership. The model call is the easy part. The hard part is everything around it — the same hard part you already know from Kubernetes platforms, data platforms, and shared services.

If you only remember four moves, remember these:

  1. Call models with workload identity through a controlled gateway.
  2. Constrain what can be deployed with policy, not heroics.
  3. Put hub, project, deployment, and app wiring in GitHub-backed IaC.
  4. Promote changes with evaluation and audit evidence, not portal clicks.

Do that, and “we shipped a copilot” becomes “we run an AI platform.” The first is a demo. The second is an organization capability.