Instruction Tuning and SFT

"Instruction tuning turns a language model into a cooperative interface."

Overview

Supervised fine-tuning aligns a pretrained next-token model with demonstrated instruction-following behavior by optimizing response tokens under a curated chat protocol.

Alignment and safety sit between model training and model deployment. The chapter studies how desired behavior is specified, learned, stress-tested, constrained, and improved through feedback.

This section is written in LaTeX Markdown. Inline mathematics uses $...$, and display equations use `

`. The notes use math to expose the operational contract: which behavior is optimized, which behavior is forbidden, and which evidence proves the system changed.

Prerequisites

• Fine-Tuning Math
• Cross-Entropy
• Data Format Standards
• Capability Benchmarks

Companion Notebooks

Learning Objectives

After completing this section, you will be able to:

• Define the core mathematical objects in instruction tuning and sft
• Write alignment losses and decision rules in repo notation
• Separate data, objective, policy, and evaluation responsibilities
• Explain how safety failures become training or guardrail updates
• Identify reward hacking, over-refusal, proxy misspecification, and drift
• Design synthetic experiments that demonstrate alignment tradeoffs
• Connect human feedback to SFT, reward modeling, DPO, or runtime policies
• State the boundary with Chapter 17 evaluation and Chapter 19 production MLOps
• Read modern alignment papers with attention to assumptions and proxies
• Build exercises that test both formulas and system-level judgment

Study Flow

1. Read the pages in order and pause after each page to restate the main definition or theorem.
2. Run theory.ipynb when you want to check the formulas numerically.
3. Use exercises.ipynb after the reading path, not before it.
4. Return to this overview page when you need the chapter-level navigation.

Runnable Companions

• Theory notebook
• Exercises notebook