Hugging Face Transformers on Dedicated GPU

You will deploy Hugging Face Transformers models on a dedicated GPU server with optimised inference settings. By the end of this guide, you will have models loaded efficiently on GPU, quantised for memory savings, and served through a production-ready API.

Environment Setup

Install the core libraries. Transformers needs PyTorch with CUDA support and the Accelerate library for multi-GPU and memory management.

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers accelerate bitsandbytes

# Verify GPU access
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"VRAM: {torch.cuda.get_device_properties(0).total_mem / 1e9:.1f} GB")

For CUDA troubleshooting, see our CUDA OOM fix guide. Ensure your PyTorch installation matches your driver version.

Model Loading Strategies

How you load a model determines VRAM usage and inference speed. Choose the strategy based on your GPU capacity and latency requirements.

from transformers import AutoModelForCausalLM, AutoTokenizer

# Standard FP16 loading
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

# 4-bit quantised loading (saves ~60% VRAM)
model_4bit = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    device_map="auto"
)

# 8-bit loading (saves ~40% VRAM, better quality than 4-bit)
model_8bit = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    load_in_8bit=True,
    device_map="auto"
)

The device_map="auto" parameter uses Accelerate to distribute model layers across available GPUs. For single-GPU setups, it loads everything onto GPU 0.

Running Inference

The pipeline API provides the simplest interface. For fine-grained control, use the model and tokeniser directly.

from transformers import pipeline

# Pipeline API (simplest)
generator = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    device_map="auto"
)

result = generator(
    "Explain GPU memory hierarchy in two sentences.",
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True
)
print(result[0]["generated_text"])

# Direct model usage (more control)
inputs = tokenizer("Explain CUDA cores:", return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        do_sample=True,
        top_p=0.9
    )
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Inference Optimisation

Squeeze more throughput from your GPU with these optimisation techniques.

# Enable Flash Attention 2 (requires compatible GPU)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype=torch.float16,
    attn_implementation="flash_attention_2",
    device_map="auto"
)

# Use BetterTransformer for older models
from optimum.bettertransformer import BetterTransformer
model = BetterTransformer.transform(model)

# Compile with torch.compile for repeated inference
model = torch.compile(model, mode="reduce-overhead")

Flash Attention 2 reduces memory usage and increases speed for long sequences. For production inference at scale, consider vLLM which includes these optimisations plus continuous batching. See the vLLM setup guide for migration.

Production Serving

Wrap your model in a FastAPI server for HTTP access. Keep the model loaded in memory and handle requests concurrently.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)

class GenerateRequest(BaseModel):
    prompt: str
    max_tokens: int = 256

@app.post("/generate")
async def generate(req: GenerateRequest):
    result = generator(req.prompt, max_new_tokens=req.max_tokens)
    return {"text": result[0]["generated_text"]}

For higher throughput, switch to vLLM or use the Transformers model inside a Celery worker for async processing. Monitor GPU metrics with Prometheus and Grafana. The self-hosting guide covers broader deployment patterns, and our tutorials section has additional examples.