LLM Context Window: Sliding Strategy

Your Conversation Exceeds the Model’s Context Window

After 15 turns of conversation, your LLM API returns an error: the input exceeds the model’s maximum context length. Or worse, it silently truncates early messages and the model loses track of what the user said at the beginning. A Llama 3.1 8B with an 8K context fills up quickly once you add a system prompt, conversation history, and a document for reference. Managing context on your GPU server requires a deliberate strategy — not just hoping conversations stay short.

Accurate Token Counting

Before implementing a sliding window, count tokens accurately for your specific model:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")

def count_tokens(messages):
    """Count tokens for a chat message list."""
    total = 0
    for msg in messages:
        # Each message has role/content overhead (model-specific)
        total += len(tokenizer.encode(msg["content"]))
        total += 4  # Approximate overhead for role tags
    total += 2  # Start/end tokens
    return total

# Example
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me about GPU computing."},
    {"role": "assistant", "content": "GPU computing uses parallel..."},
]
print(f"Token count: {count_tokens(messages)}")

# Budget allocation for 8192-token context:
# System prompt:     ~500 tokens (fixed)
# Max response:     ~1000 tokens (reserved for generation)
# Conversation:     ~6692 tokens (available for history)

Basic Sliding Window

Drop the oldest messages when the conversation exceeds the token budget:

def sliding_window(messages, max_tokens=8192, reserved_for_response=1000):
    available = max_tokens - reserved_for_response

    # Always keep the system message
    system_msgs = [m for m in messages if m["role"] == "system"]
    conversation = [m for m in messages if m["role"] != "system"]

    system_tokens = count_tokens(system_msgs)
    budget = available - system_tokens

    # Keep messages from newest to oldest until budget exhausted
    kept = []
    running_tokens = 0
    for msg in reversed(conversation):
        msg_tokens = count_tokens([msg])
        if running_tokens + msg_tokens > budget:
            break
        kept.insert(0, msg)
        running_tokens += msg_tokens

    return system_msgs + kept

# Usage
trimmed = sliding_window(full_conversation, max_tokens=8192)
response = call_llm(trimmed)

Smart Truncation with Summarisation

Instead of dropping old messages entirely, summarise them to preserve context:

async def summarise_and_slide(messages, max_tokens=8192):
    system_msgs = [m for m in messages if m["role"] == "system"]
    conversation = [m for m in messages if m["role"] != "system"]

    if count_tokens(messages) <= max_tokens:
        return messages  # No truncation needed

    # Split conversation into old and recent halves
    midpoint = len(conversation) // 2
    old_messages = conversation[:midpoint]
    recent_messages = conversation[midpoint:]

    # Summarise old messages using the LLM itself
    summary_prompt = [
        {"role": "system", "content": "Summarise this conversation in 2-3 sentences. "
         "Include key facts, decisions, and user preferences."},
        *old_messages
    ]
    summary = await call_llm(summary_prompt, max_tokens=200)

    # Reconstruct with summary replacing old messages
    return system_msgs + [
        {"role": "system", "content": f"Previous conversation summary: {summary}"},
        *recent_messages
    ]

Priority-Based Message Retention

Not all messages are equally important. Keep high-value messages while dropping filler:

def priority_window(messages, max_tokens=8192, reserved=1000):
    system_msgs = [m for m in messages if m["role"] == "system"]
    conversation = [m for m in messages if m["role"] != "system"]
    budget = max_tokens - reserved - count_tokens(system_msgs)

    # Assign priority scores
    scored = []
    for i, msg in enumerate(conversation):
        score = 0
        score += 10 if i >= len(conversation) - 4 else 0  # Recent messages
        score += 5 if msg["role"] == "user" else 3         # User msgs over assistant
        score += 3 if "?" in msg["content"] else 0         # Questions
        score += 2 if len(msg["content"]) > 200 else 0     # Substantial messages
        scored.append((score, i, msg))

    # Sort by priority, take highest-scoring messages within budget
    scored.sort(key=lambda x: (-x[0], x[1]))
    kept = []
    tokens_used = 0
    for score, idx, msg in scored:
        msg_tokens = count_tokens([msg])
        if tokens_used + msg_tokens <= budget:
            kept.append((idx, msg))
            tokens_used += msg_tokens

    # Restore chronological order
    kept.sort(key=lambda x: x[0])
    return system_msgs + [msg for _, msg in kept]

Putting It Together in Production

class ConversationManager:
    def __init__(self, model_max_tokens=8192, response_reserve=1000):
        self.max_tokens = model_max_tokens
        self.reserve = response_reserve
        self.conversations = {}  # session_id -> messages

    async def add_and_generate(self, session_id, user_message):
        if session_id not in self.conversations:
            self.conversations[session_id] = [
                {"role": "system", "content": SYSTEM_PROMPT}
            ]

        self.conversations[session_id].append(
            {"role": "user", "content": user_message})

        # Apply sliding window if needed
        messages = await summarise_and_slide(
            self.conversations[session_id], self.max_tokens)

        response = await call_llm(messages, max_tokens=self.reserve)
        self.conversations[session_id].append(
            {"role": "assistant", "content": response})

        return response

Context window management is critical for vLLM and Ollama deployments on your GPU server. The vLLM production guide covers max-model-len configuration. See the LLM hosting section for architecture patterns, tutorials for implementation guides, and benchmarks for context-length performance data.