LLM Temperature & Sampling Config Guide

Your LLM Outputs Are Either Boring or Incoherent

At temperature 0.1, every response reads identically — safe, formulaic, lifeless. At temperature 1.5, the model produces creative gibberish with made-up words and broken grammar. Finding the right sampling configuration is the difference between a production-quality LLM and an unreliable one. These parameters control how the model selects tokens from its probability distribution, and misconfiguring them on your GPU server is one of the most common causes of poor output quality.

Temperature: What It Actually Does

Temperature scales the logit values before the softmax function, sharpening or flattening the probability distribution:

# Temperature = 0.0 (greedy): always picks the highest-probability token
# Temperature = 0.7: moderately sharpens probabilities (good default)
# Temperature = 1.0: uses raw model probabilities
# Temperature = 1.5+: flattens probabilities (more random, often incoherent)

# vLLM configuration
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Explain gravity"}],
    "temperature": 0.7
  }'

# Ollama configuration
curl http://localhost:11434/api/generate \
  -d '{
    "model": "llama3.1:8b",
    "prompt": "Explain gravity",
    "options": {"temperature": 0.7}
  }'

Top-p and Top-k: Truncating the Distribution

These parameters remove low-probability tokens before sampling, preventing the model from selecting nonsensical continuations:

# Top-p (nucleus sampling): keep tokens whose cumulative probability <= p
# Top-p = 0.9: considers tokens covering 90% of probability mass
# Top-p = 1.0: no filtering (all tokens eligible)
# Top-p = 0.1: only the most likely tokens (very conservative)

# Top-k: keep only the k highest-probability tokens
# Top-k = 50: standard filtering
# Top-k = 1: equivalent to greedy decoding
# Top-k = -1 or 0: disabled (all tokens eligible)

# Combined usage in vLLM
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Write a poem"}],
    "temperature": 0.8,
    "top_p": 0.9,
    "top_k": 50
  }'

# Important: top-p and top-k interact with temperature
# High temperature + high top-p = very random output
# Low temperature + low top-p = very deterministic output

Repetition and Frequency Penalties

Without penalties, LLMs fall into repetitive loops. These parameters discourage repeated tokens:

# repetition_penalty: penalises tokens that have already appeared
# 1.0 = no penalty, 1.1 = mild penalty, 1.3 = strong penalty
# Values above 1.5 can cause incoherent output

# frequency_penalty: penalises based on how often a token has appeared
# 0.0 = no penalty, 0.5 = moderate, 1.0 = strong

# presence_penalty: penalises any token that has appeared at all
# 0.0 = no penalty, 0.5 = moderate, 1.0 = strong

# vLLM supports all three
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "List ten creative ideas"}],
    "temperature": 0.8,
    "top_p": 0.95,
    "frequency_penalty": 0.3,
    "presence_penalty": 0.3
  }'

Recommended Presets by Use Case

# Factual Q&A / Code Generation / Data Extraction
temperature: 0.0-0.2
top_p: 1.0
repetition_penalty: 1.0
# Rationale: deterministic, consistent, no creativity needed

# Customer Support / Technical Writing
temperature: 0.4-0.6
top_p: 0.9
repetition_penalty: 1.05
# Rationale: slight variation while staying on-topic

# Creative Writing / Brainstorming
temperature: 0.7-0.9
top_p: 0.95
frequency_penalty: 0.3
presence_penalty: 0.3
# Rationale: diverse output without losing coherence

# Roleplay / Fiction / Poetry
temperature: 0.8-1.0
top_p: 0.95
top_k: 100
frequency_penalty: 0.5
# Rationale: maximum creativity with guardrails

# Translation
temperature: 0.1-0.3
top_p: 0.9
repetition_penalty: 1.0
# Rationale: accuracy over creativity

Testing and Validating Your Configuration

Run the same prompt multiple times at each configuration to evaluate consistency and quality:

import requests, json

configs = [
    {"temperature": 0.0, "top_p": 1.0},
    {"temperature": 0.5, "top_p": 0.9},
    {"temperature": 0.8, "top_p": 0.95},
]

prompt = "Explain how neural networks learn, in two sentences."

for cfg in configs:
    results = []
    for _ in range(5):
        r = requests.post("http://localhost:8000/v1/chat/completions",
            json={"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
                  "messages": [{"role": "user", "content": prompt}],
                  **cfg})
        results.append(r.json()["choices"][0]["message"]["content"])
    unique = len(set(results))
    print(f"Config {cfg}: {unique}/5 unique responses")

For vLLM and Ollama deployments on your GPU server, set sampling defaults at the server level and allow per-request overrides. The vLLM production guide covers API configuration. Check the LLM hosting section for deployment patterns, benchmarks for throughput data, and our tutorials for step-by-step guides.