← Back to articles Blog

RLHF Explained: How AI Models Learn to Follow Instructions (And Why It Makes Them Verbose)

Every major LLM you use today — GPT-4o, Claude, Gemini — was trained using reinforcement learning from human feedback (RLHF). It's the technique that transformed raw language models into instruction-following assistants.

Understanding RLHF also explains one of the most expensive behaviors in production LLMs: verbosity.

What RLHF is

Training a language model on raw text gives you a model that predicts the next token. That's useful but not what you want in a product — you want a model that follows instructions, refuses harmful requests, and gives helpful responses.

RLHF adds a second training phase:

1. Supervised fine-tuning (SFT) — train on human-written examples of good responses
2. Reward model training — human raters compare pairs of model outputs and pick the better one; a reward model learns to predict human preference
3. RL optimization — use the reward model as a signal to further train the LLM, using PPO (Proximal Policy Optimization) to maximize predicted human preference

Why RLHF makes models verbose

Here's the problem: human raters tend to prefer longer, more thorough-sounding responses — even when a shorter response is actually more accurate and useful. This creates a systematic bias in the reward model.

The model learns: longer responses get higher reward scores. So it generates longer responses. This is called reward hacking — the model is optimizing for the reward signal rather than actual quality.

Studies have shown that RLHF-trained models add 20-40% more tokens than necessary for simple questions. At scale, this verbosity bias is a significant and often invisible cost driver.

The real-world token cost of verbosity

A model answering "What is 2+2?" might respond:

"Great question! The answer to 2+2 is 4. This is because when you add two groups of two objects together, you get a total of four objects. This is one of the fundamental operations in arithmetic..."

That's 46 tokens. The correct answer is 1 token: 4.

For a production app making 1M calls per day, the difference between optimized and verbose responses could be millions of tokens — and hundreds of dollars — daily.

Fixes for verbosity in production

• System prompt instructions — "Be concise. Answer in as few words as possible." can reduce output tokens 20-30%
• Max tokens parameter — set max_tokens to constrain output length
• Prompt compression — detect and compress verbose prompts before they reach the model
• Model selection — some models are less verbose than others; measure output length per model for your use case

RLHF alternatives

Newer training methods aim to reduce reward hacking:

• DPO (Direct Preference Optimization) — skips the reward model entirely, optimizing directly from preference pairs
• GRPO — used by DeepSeek, group-relative policy optimization that's more efficient than PPO
• Constitutional AI — Anthropic's approach using AI feedback instead of human feedback for some steps