Connect AWS S3 to GPU for Models

What You’ll Connect

After this guide, your GPU server will pull AI models directly from AWS S3 buckets — syncing trained models from your cloud training pipelines to your dedicated GPU hardware for production inference. Your vLLM or Ollama endpoint loads models from a local cache that stays synchronised with your S3 model registry, bridging cloud training with on-premises inference.

The integration uses the AWS CLI and boto3 for efficient model transfers. S3 Transfer Acceleration speeds up large model downloads, multi-part transfers handle 100GB+ model files reliably, and incremental sync ensures only changed files transfer when a model updates.

Prerequisites

• A GigaGPU server with 200GB+ storage for model files
• An AWS account with S3 access and IAM credentials
• AWS CLI v2 installed on the GPU server
• A running inference endpoint (vLLM production guide)

Integration Steps

Configure AWS credentials on your GPU server using aws configure or environment variables. Use an IAM role with read-only access to your model bucket — the GPU server should not have write permissions to your training data. Set the AWS region to match your S3 bucket for optimal transfer speeds.

Organise your S3 bucket with a clear model registry structure: s3://your-models/model-name/version/. Training pipelines upload completed model checkpoints to versioned paths. A metadata file at the bucket root tracks which version is current for each model, so your sync script knows which version to download.

Build a sync script that reads the metadata file, compares against locally cached versions, and downloads the latest model using aws s3 sync for efficient incremental transfers. Schedule the sync as a cron job or trigger it on demand when you promote a new model version.

Code Example

Model sync pipeline between AWS S3 and your GPU inference server:

import boto3, json, os, subprocess

BUCKET = "your-ai-models"
LOCAL_CACHE = "/data/models"
METADATA_KEY = "model-registry.json"

s3 = boto3.client("s3")

def get_registry():
    """Fetch model registry metadata from S3."""
    obj = s3.get_object(Bucket=BUCKET, Key=METADATA_KEY)
    return json.loads(obj["Body"].read())

def sync_model(model_name, version):
    """Sync a specific model version from S3 to local cache."""
    s3_path = f"s3://{BUCKET}/{model_name}/{version}/"
    local_path = os.path.join(LOCAL_CACHE, model_name)
    os.makedirs(local_path, exist_ok=True)

    cmd = [
        "aws", "s3", "sync", s3_path, local_path,
        "--only-show-errors",
        "--no-progress"
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        raise RuntimeError(f"S3 sync failed: {result.stderr}")
    return local_path

def sync_all_production_models():
    """Sync all production models from the registry."""
    registry = get_registry()
    updated = []

    for model_name, config in registry["models"].items():
        version = config["production_version"]
        version_file = os.path.join(LOCAL_CACHE, model_name, ".version")
        current = open(version_file).read().strip() \
            if os.path.exists(version_file) else ""

        if version != current:
            print(f"Syncing {model_name} v{version}...")
            sync_model(model_name, version)
            with open(version_file, "w") as f:
                f.write(version)
            updated.append(model_name)

    if updated:
        print(f"Updated: {updated}. Restarting inference server...")
        subprocess.run(["docker", "restart", "vllm-inference"])
    else:
        print("All models up to date.")

if __name__ == "__main__":
    sync_all_production_models()

Testing Your Integration

Upload a small test model to your S3 bucket with the expected directory structure. Update the registry metadata file with the test model entry. Run the sync script and verify files download to the correct local cache directory. Run the sync again to confirm incremental behaviour — unchanged files should not re-download.

Test the version update flow: upload a new version, update the registry, run sync, and verify vLLM loads the new model via the OpenAI-compatible API. Measure download speeds to ensure the S3 region and network configuration are optimal.

Production Tips

Enable S3 Transfer Acceleration for faster downloads, especially when the GPU server is geographically distant from the S3 bucket region. Use S3 lifecycle policies to transition old model versions to Glacier storage after 90 days, keeping costs low while maintaining rollback capability. Set up S3 event notifications with SNS to trigger automatic sync when a new model version is uploaded.

For security, use IAM roles with minimal permissions — read-only on the model bucket. Encrypt model files at rest with S3 server-side encryption and in transit with TLS. For multi-GPU deployments, each server syncs independently from S3, ensuring consistent model versions across your fleet. Explore more tutorials or get started with GigaGPU for your open-source model infrastructure.