LoRA Loading Errors in Stable Diffusion: Fix

Symptom: LoRA Fails to Load or Has No Effect

You downloaded a LoRA from CivitAI or trained one yourself, tried to load it into your Stable Diffusion pipeline on your GPU server, and got one of these:

ValueError: The following keys have not been correctly renamed: lora_unet_down_blocks_0_attentions_0_transformer_blocks_0...

RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel: size mismatch for...

Or the LoRA loads without errors but the generated images look identical to the base model, as if the LoRA has zero effect. Both scenarios come down to format incompatibilities, incorrect loading methods, or wrong conditioning scale.

Fix 1: Use the Correct Loading Method

LoRA files come in multiple formats. The loading method must match:

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

# For diffusers-format LoRA (from Hugging Face)
pipe.load_lora_weights("username/lora-name")

# For single-file LoRA (from CivitAI or Kohya)
pipe.load_lora_weights(".", weight_name="my-lora.safetensors")

# For A1111-format LoRA files
pipe.load_lora_weights(
    ".",
    weight_name="my-lora.safetensors",
)

The weight_name parameter is essential when loading from a local file rather than a Hugging Face repository.

Fix 2: Match LoRA to Base Model Architecture

A LoRA trained on SD 1.5 will not work with SDXL and vice versa:

# Check LoRA architecture by inspecting its keys
from safetensors.torch import load_file

lora = load_file("my-lora.safetensors")
keys = list(lora.keys())
print(f"Total keys: {len(keys)}")
print("Sample keys:", keys[:5])

# SD 1.5 LoRA keys typically contain:
#   lora_unet_down_blocks_...
#   lora_te_text_model_encoder_...

# SDXL LoRA keys typically contain:
#   lora_unet_down_blocks_... (more blocks)
#   lora_te1_... and lora_te2_... (two text encoders)

# Use the correct pipeline for the LoRA's architecture
from diffusers import StableDiffusionXLPipeline
pipe_xl = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16
).to("cuda")
pipe_xl.load_lora_weights(".", weight_name="sdxl-lora.safetensors")

Fix 3: Adjust the LoRA Scale

If the LoRA loads but has no visible effect, the conditioning scale may be too low:

# Default scale is 1.0, but some LoRAs need adjustment
pipe.load_lora_weights(".", weight_name="style-lora.safetensors")

# Generate with explicit scale
image = pipe(
    "a portrait in the trained style",
    num_inference_steps=25,
    cross_attention_kwargs={"scale": 1.0}  # Range: 0.0 to 2.0
).images[0]

# Higher scale = stronger LoRA effect (may cause artifacts above 1.5)
# Lower scale = subtler effect
# Try 0.7 to 1.2 for most LoRAs

Fix 4: Stack Multiple LoRAs

Combining LoRAs requires careful weight management:

# Load first LoRA
pipe.load_lora_weights(".", weight_name="style-lora.safetensors",
                       adapter_name="style")

# Load second LoRA
pipe.load_lora_weights(".", weight_name="character-lora.safetensors",
                       adapter_name="character")

# Set weights for each adapter
pipe.set_adapters(["style", "character"], adapter_weights=[0.8, 0.6])

# Generate with both active
image = pipe("a character in the style", num_inference_steps=25).images[0]

# Disable a specific adapter
pipe.set_adapters(["style"], adapter_weights=[1.0])

# Remove all LoRAs
pipe.unload_lora_weights()

Fix 5: Fuse LoRA for Production Speed

Fusing the LoRA weights into the base model eliminates the overhead of dynamic weight merging:

# Fuse LoRA into the model weights permanently
pipe.load_lora_weights(".", weight_name="my-lora.safetensors")
pipe.fuse_lora(lora_scale=0.8)

# Now generate without cross_attention_kwargs
image = pipe("prompt", num_inference_steps=25).images[0]

# Unfuse to restore the original model
pipe.unfuse_lora()

Fused models run at the same speed as the base model with no per-token LoRA overhead. For Stable Diffusion hosting workflows involving many LoRAs, ComfyUI handles switching and stacking through its node interface. Check the PyTorch guide for PyTorch version requirements, the benchmarks for LoRA overhead measurements, and the tutorials section for training your own LoRAs. Our CUDA guide covers the driver setup.