Fine-tuning & Model Customization

Before you start: this is a billed Azure service

• Cost and account. An Azure subscription and a Foundry project. Fine-tuning is one of the more expensive operations in this series — training jobs and the resulting hosted model both bill. Use a tiny dataset and delete the model when done.
• Authored against the docs, not executed live. Code is written against the official docs and syntax-checked, not run during authoring. Pin your SDK version and verify.
• Two languages. Python-primary (OpenAI-compatible fine-tuning) with .NET noted.

About this series

This is Part 11 of the twelve-part Microsoft Foundry series. You've made the agent better with prompting, grounding (Lab 3), tools, and memory. This lab covers the heaviest lever — customizing the model itself — and, just as important, when not to.

What you'll build

You'll run a model customization end to end and learn to judge when it's worth it:

1. Decide whether to customize at all — the fine-tune-vs-ground-vs-prompt question.
2. Prepare a small training dataset in the chat format.
3. Launch a fine-tuning job and track it.
4. Deploy and evaluate the customized model against the base — does it actually win?

You'll also meet distillation and reinforcement fine-tuning (RFT) as options on the spectrum.

Why this matters (the 5-minute business case)

Read this before handing the lab to your team.

Fine-tuning is powerful and frequently the wrong first move. Teams reach for it to "teach the model our business," but most "it doesn't know our stuff" problems are knowledge problems (solved by grounding) or behavior problems (solved by prompting and examples). Fine-tuning earns its cost in a narrower set of cases — and knowing which is a money decision.

Three reasons this is a leadership-level concern:

1. Customization can buy consistency, smaller models, and lower latency. When you need a behavior reliably at scale that prompting can't pin down, or you want a smaller/cheaper model to match a bigger one on a narrow task, fine-tuning (or distillation) can pay off — sometimes dramatically on per-call cost.
2. It's also the most expensive and brittle lever. Training costs money, the result needs hosting (more cost), and it goes stale as your domain changes — you retrain. Grounding updates with a document edit; a fine-tune updates with a training run. Choose deliberately.
3. RFT is a new tool for hard tasks. GPT-5 Reinforcement Fine-Tuning lets you train on domain-specific tasks with reinforcement signals — useful for complex, gradeable problems where plain supervised fine-tuning falls short. Powerful, and the most involved option.

The decision rule: reach for prompting/examples for behavior, grounding for knowledge, and fine-tuning only when you need a consistent behavior at scale that the cheaper levers can't deliver — or a smaller/faster model for a narrow task. Measure the win against the base model (Lab 6) before you commit.

A framing for a non-technical stakeholder: "Fine-tuning customizes the model itself. It's powerful but expensive and needs maintenance, so we use it only when cheaper options — better prompts, grounding in our documents — can't get there, and we prove it beats the base model first."

Concepts you'll use (2-minute primer)

• Fine-tuning — supervised training on your examples to shape behavior/format consistently.
• Distillation — train a smaller model to mimic a larger one on your task, for cheaper/faster inference.
• Reinforcement fine-tuning (RFT) — train with reinforcement signals on gradeable tasks (e.g., GPT-5 RFT) for hard problems.
• Training dataset — examples in the chat (messages) format: input + the ideal output.
• The OpenAI-compatible client — Foundry exposes fine-tuning through the OpenAI-style client (.get_openai_client()), so the workflow is familiar and portable.

One sentence: customize the model only when prompting and grounding can't deliver the behavior or cost you need — and prove the customized model beats the base before shipping it.

Prerequisites

Step 1 — Prepare a training dataset

Fine-tuning data is examples in the chat format — input plus the ideal response. Quality and consistency matter far more than volume:

{"messages": [{"role": "system", "content": "You are the Acme Orders assistant."}, {"role": "user", "content": "Can I return opened headphones after 20 days?"}, {"role": "assistant", "content": "Opened electronics have a 15-day return window, so 20 days is past it. I can help with an exchange instead."}]}
{"messages": [{"role": "system", "content": "You are the Acme Orders assistant."}, {"role": "user", "content": "Where's my order?"}, {"role": "assistant", "content": "Happy to help! What's your order ID? It's on your confirmation email as AC-#####."}]}

Every example should model the exact behavior you want, in the tone you want. Garbage examples train garbage.

Step 2 — Launch a fine-tuning job

Foundry exposes fine-tuning through the OpenAI-compatible client:

import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

project = AIProjectClient(
    endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
    credential=DefaultAzureCredential(),
)
client = project.get_openai_client()

# Upload the training file, then create the job.
training = client.files.create(file=open("acme_finetune.jsonl", "rb"), purpose="fine-tune")
job = client.fine_tuning.jobs.create(
    training_file=training.id,
    model="gpt-5-mini",  # a fine-tunable base model
)
print("job:", job.id)

# Poll status until it completes:
print(client.fine_tuning.jobs.retrieve(job.id).status)

Training takes a while (and bills). When it finishes you get a fine-tuned model you then deploy.

The fine-tuning surface (and RFT/distillation options) varies by base model and SDK version — check the catalog and docs for what your chosen model supports.

.NET note: fine-tuning is driven the same way via the OpenAI-compatible client in C#; the job lifecycle mirrors Python.

Step 3 — Deploy and evaluate against the base

A customized model is only worth it if it beats the base. Deploy the fine-tuned model, then run the evaluation loop from Lab 6 comparing it against the base model on your real dataset:

• Did quality measurably improve on the target behavior?
• Did per-call cost or latency improve (the usual reason to distill to a smaller model)?
• Did anything regress elsewhere?

If the customized model doesn't clearly win on a metric you care about, don't ship it — go back to prompting and grounding. This is the discipline that keeps fine-tuning from becoming an expensive habit.

Cost & cleanup

Fine-tuning is among the costliest things in this series — clean up promptly:

• Delete the deployed fine-tuned model when done (it bills while hosted).
• Delete training files and jobs artifacts you don't need.
• Confirm in Cost Management that nothing is still accruing.

If a lab created it and you're done, delete it today.

Verified against

• Microsoft Foundry fine-tuning / distillation / GPT-5 RFT docs and the OpenAI-compatible fine-tuning surface via `azure-ai-projects`, June 2026.
• Fine-tuning support, options, and method names vary by base model and SDK version; code is syntax-checked against the docs but not executed during authoring. Pin your version.

What's next

You've seen the full toolkit — and when to use the heaviest lever. One lab remains:

• Lab 12 — Interoperability capstone: tie a Foundry agent to the open ecosystem with MCP and A2A, and publish it to Microsoft 365 — managed and open.