Temperature in LLMs: What It Is and How to Set It

Temperature is a sampling parameter that controls how random or deterministic an LLM's output is. It's one of the most important parameters you can set — and one of the most misunderstood. Temperature directly affects output quality, consistency, and token cost.

What temperature actually does

When an LLM generates text, it produces a probability distribution over all possible next tokens at each step. Temperature scales this distribution before sampling:

• Low temperature (0.0–0.3) — makes the distribution sharper. The model almost always picks the highest-probability token. Output is deterministic, consistent, and focused.
• Temperature 1.0 — the model samples from its raw probability distribution. This is the default for most providers.
• High temperature (1.5–2.0) — flattens the distribution. The model is more likely to pick lower-probability tokens. Output is more varied, creative, and sometimes incoherent.

The mathematical intuition

Temperature T divides the logits (raw scores) before the softmax function converts them to probabilities. When T approaches 0, the highest-scoring token gets probability approaching 1. When T is very high, all tokens approach equal probability — effectively random selection.

temperature = 0.0  # Always picks the most likely token
temperature = 1.0  # Samples from the raw distribution
temperature = 2.0  # Much more random — low-probability tokens become likely

Temperature by use case

Temperature and token cost

Temperature has a subtle but real effect on token cost. Higher temperatures tend to produce longer outputs — because the model is more likely to continue generating when it would otherwise stop. A verbose, high-temperature response uses more output tokens than a focused, low-temperature one for the same prompt.

For high-volume production workloads where output tokens dominate cost, setting temperature to 0.0–0.3 and adding explicit length constraints is one of the cheapest optimizations available. Combined with prompt compression and semantic caching, it compounds into meaningful savings.

Temperature vs top_p: what's the difference

Top-p (nucleus sampling) is an alternative to temperature that also controls output randomness. Instead of scaling probabilities, it samples from the smallest set of tokens whose cumulative probability exceeds p.

Most practitioners use temperature and leave top_p at its default (1.0). Adjusting both simultaneously is generally not recommended — they interact in ways that are hard to predict.

Setting temperature in the API

import { wrap } from 'tokoscope'
import OpenAI from 'openai'

const client = wrap(new OpenAI(), { apiKey: 'ts_live_...' })

// Deterministic code generation
const code = await client.chat.completions.create({
  model: 'gpt-4o',
  temperature: 0.0,
  messages: [{ role: 'user', content: 'Write a binary search function in Python' }]
})

// Creative writing
const story = await client.chat.completions.create({
  model: 'gpt-4o',
  temperature: 1.1,
  messages: [{ role: 'user', content: 'Write the opening paragraph of a sci-fi story' }]
})

Temperature 0 for tasks with a right answer. Temperature 1+ for tasks where variety is the point. Everything else in between based on how much consistency vs creativity you need.