Unsloth Fine-Tuning on RTX 4060 Ti 16GB

Unsloth is a fine-tuning library with custom Triton kernels that cut memory use and boost speed by 2-3x over standard QLoRA. On a 16 GB RTX 4060 Ti from our dedicated GPU hosting, it turns a tight training budget into a comfortable one.

Contents

• Installation
• Minimal training script
• Speed numbers
• Limits

Install

pip install "unsloth[cu124-torch230] @ git+https://github.com/unslothai/unsloth.git"

Pick the wheel matching your CUDA / PyTorch. Unsloth’s docs list exact combinations.

Training

from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.1-8b-Instruct",
    max_seq_length=4096,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16, lora_alpha=32,
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    use_gradient_checkpointing="unsloth",
)

cfg = SFTConfig(
    output_dir="./out",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    bf16=True,
    num_train_epochs=3,
    optim="adamw_8bit",
)
SFTTrainer(model=model, tokenizer=tokenizer, args=cfg, train_dataset=your_dataset).train()

Speed

On a 4060 Ti 16 GB fine-tuning Llama 3.1 8B:

Roughly 2-2.5x over stock PEFT. On small GPUs that is the difference between an hour-long run and a three-hour run.

Limits

Unsloth supports a specific subset of architectures (Llama, Mistral, Gemma, Phi, Qwen, and close derivatives). Exotic models – novel architectures, MoE variants, custom tokenisers – may not work. For broader compatibility fall back to Axolotl or raw PEFT.

See LoRA on Mistral 7B and Axolotl.