OpenAI SDK with Self-Hosted Models: Node.js Guide

You will use the official OpenAI Node.js SDK to connect to self-hosted models running on your own GPU server. By the end of this guide, you will have working TypeScript examples for chat completions, streaming, function calling, and integration with Express — all hitting a local inference endpoint with no per-token costs.

Installation and Configuration

Install the OpenAI SDK and point it at your self-hosted backend. Both vLLM and Ollama expose OpenAI-compatible endpoints that the SDK connects to without modification.

npm install openai

// client.ts
import OpenAI from "openai";

// For vLLM backend
const client = new OpenAI({
  baseURL: "http://localhost:8000/v1",
  apiKey: "not-needed",
});

// For Ollama backend
const ollamaClient = new OpenAI({
  baseURL: "http://localhost:11434/v1",
  apiKey: "ollama",
});

The SDK interface is identical regardless of backend. Only the baseURL and model name change. For backend setup, see the vLLM production guide.

Chat Completions

The chat completions endpoint works identically to the OpenAI hosted service. Request and response shapes match the official TypeScript types.

async function chat(userMessage: string): Promise<string> {
  const response = await client.chat.completions.create({
    model: "meta-llama/Llama-3.1-8B-Instruct",
    messages: [
      { role: "system", content: "You are a concise technical assistant." },
      { role: "user", content: userMessage },
    ],
    max_tokens: 512,
    temperature: 0.7,
  });

  return response.choices[0].message.content ?? "";
}

const answer = await chat("What is KV-cache in transformer inference?");
console.log(answer);

For the Python equivalent, see the Python SDK guide. The API compatibility guide covers which endpoints vLLM supports.

Streaming Responses

Token-by-token streaming is critical for chat interfaces. The Node.js SDK provides an async iterator that works with self-hosted models exactly as it does with OpenAI.

async function streamChat(userMessage: string): Promise<string> {
  const stream = await client.chat.completions.create({
    model: "meta-llama/Llama-3.1-8B-Instruct",
    messages: [{ role: "user", content: userMessage }],
    max_tokens: 1024,
    stream: true,
  });

  let collected = "";
  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) {
      process.stdout.write(content);
      collected += content;
    }
  }
  return collected;
}

await streamChat("Explain tensor parallelism step by step.");

For building a full-stack streaming UI, see the Next.js integration guide or the React chat UI guide.

Function Calling

Models with tool-use training support OpenAI-format function calling. Define tools as JSON Schema and the model determines when to invoke them.

const tools: OpenAI.Chat.Completions.ChatCompletionTool[] = [
  {
    type: "function",
    function: {
      name: "get_gpu_stats",
      description: "Get current GPU utilisation and memory usage",
      parameters: {
        type: "object",
        properties: {
          gpu_id: { type: "number", description: "GPU device index" },
        },
        required: ["gpu_id"],
      },
    },
  },
];

const response = await client.chat.completions.create({
  model: "meta-llama/Llama-3.1-8B-Instruct",
  messages: [{ role: "user", content: "Check the stats for GPU 0." }],
  tools,
  tool_choice: "auto",
});

const toolCall = response.choices[0].message.tool_calls?.[0];
if (toolCall) {
  console.log(`Function: ${toolCall.function.name}`);
  console.log(`Arguments: ${toolCall.function.arguments}`);
}

Express API Integration

Wrap the SDK in an Express server to build an inference API that your frontend applications consume.

import express from "express";
import OpenAI from "openai";

const app = express();
app.use(express.json());

const client = new OpenAI({
  baseURL: "http://localhost:8000/v1",
  apiKey: "not-needed",
});

app.post("/api/chat", async (req, res) => {
  const { message } = req.body;
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");

  const stream = await client.chat.completions.create({
    model: "meta-llama/Llama-3.1-8B-Instruct",
    messages: [{ role: "user", content: message }],
    stream: true,
  });

  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) {
      res.write(`data: ${JSON.stringify({ content })}\n\n`);
    }
  }
  res.write("data: [DONE]\n\n");
  res.end();
});

app.listen(3000, () => console.log("Server running on port 3000"));

This pattern gives you a streaming API endpoint that any frontend can consume via EventSource or fetch. For WebSocket alternatives, see real-time AI with WebSockets.

Error Handling and Production Tips

Self-hosted endpoints may restart during model reloads. Add retry logic for resilience.

import { APIConnectionError, APIError } from "openai";

async function generateWithRetry(prompt: string, retries = 3): Promise<string> {
  for (let attempt = 0; attempt < retries; attempt++) {
    try {
      const res = await client.chat.completions.create({
        model: "meta-llama/Llama-3.1-8B-Instruct",
        messages: [{ role: "user", content: prompt }],
        max_tokens: 256,
      });
      return res.choices[0].message.content ?? "";
    } catch (err) {
      if (err instanceof APIConnectionError && attempt < retries - 1) {
        await new Promise((r) => setTimeout(r, 2 ** attempt * 1000));
      } else {
        throw err;
      }
    }
  }
  throw new Error("Max retries exceeded");
}

Add Prometheus monitoring and structured logging with the ELK stack for production observability. The self-hosting guide covers deployment best practices, and our tutorials section has more integration examples.