How to Train an LLM: From Pre-Training to Fine-Tuning and Alignment

Training an LLM from scratch is a multi-stage process that costs tens of millions of dollars and requires clusters of thousands of GPUs. Most developers never need to do it. But understanding how training works helps you make better decisions about when to fine-tune, which alignment techniques to use, and why models behave the way they do.

The three stages of LLM training

Stage 1 — Pre-training

The base model is trained on a massive corpus of text (typically 1-15 trillion tokens) to predict the next token in a sequence. This is where the model learns language, facts, reasoning patterns, and world knowledge.

• Data: Web crawls (Common Crawl), books, code, scientific papers, filtered for quality
• Objective: Next-token prediction (cross-entropy loss)
• Scale: Thousands of GPUs for weeks to months
• Cost: $10M-$500M+ for frontier models
• Output: A base model that can complete text but doesn't follow instructions

Stage 2 — Supervised Fine-Tuning (SFT)

The pre-trained base model is fine-tuned on a curated dataset of instruction-response pairs to teach it to be a helpful assistant. This is where the model learns to follow instructions, format responses, and behave appropriately.

• Data: 10K-1M high-quality instruction-response pairs
• Objective: Next-token prediction on responses only
• Scale: Hours to days on a single GPU cluster
• Cost: $100-$50,000 depending on model size and data
• Output: An instruction-following model (chat model)

Stage 3 — Alignment (RLHF or DPO)

The SFT model is aligned with human preferences to make it helpful, honest, and safe. This is where RLHF or DPO comes in — training the model to produce responses humans rate as better over ones they rate as worse.

• Data: Preference pairs (response A vs response B ratings)
• Objective: Maximize human preference scores
• Scale: Hours to days
• Cost: $1,000-$100,000
• Output: The final deployed chat model

What most teams actually do: fine-tuning not pre-training

Starting from scratch is almost never the right choice. Pre-training requires resources only frontier labs have. Most teams start from an existing pre-trained base model and apply:

• Full fine-tuning — update all model weights on your dataset. Expensive, requires high-end GPU.
• LoRA (Low-Rank Adaptation) — train small adapter matrices, freeze most weights. 10-100x cheaper than full fine-tuning. The standard approach for most teams.
• QLoRA — LoRA on a quantized base model. Enables fine-tuning large models on consumer GPUs.

from peft import get_peft_model, LoraConfig
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-8B")
config = LoraConfig(r=8, lora_alpha=32, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, config)
# Now train normally on your dataset

When you actually need to train vs when you don't

The cost of fine-tuning in 2026

LoRA fine-tuning has become remarkably accessible:

• Llama 3.2-8B with LoRA on a single A100 80GB: ~2-4 hours, ~$20-50 on cloud GPU
• Llama 3.3-70B with QLoRA on 2x A100: ~8-16 hours, ~$100-200
• OpenAI fine-tuning (gpt-4o-mini): ~$0.008/1K training tokens, no GPU needed

The hidden cost is data preparation — high-quality instruction pairs take significant time to create or curate. Budget 10-50 hours of human effort per 1,000 high-quality examples.