vLLM Chat

vLLM can be deployed as a server that mimics the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. This server can be queried in the same format as OpenAI API.

Overview

This will help you get started with vLLM chat models, which leverages the langchain-openai package. For detailed documentation of all ChatOpenAI features and configurations head to the API reference.

Integration details

Class	Package	Local	Serializable	JS support	Downloads	Version
ChatOpenAI	langchain_openai	✅	beta	❌

Model features

Specific model features, such as tool calling, support for multi-modal inputs, support for token-level streaming, etc., will depend on the hosted model.

Setup

See the vLLM docs here.

To access vLLM models through LangChain, you’ll need to install the langchain-openai integration package.

Credentials

Authentication will depend on specifics of the inference server.

To enable automated tracing of your model calls, set your LangSmith API key:

# os.environ["LANGSMITH_TRACING"] = "true"
# os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")

Installation

The LangChain vLLM integration can be accessed via the langchain-openai package:

%pip install -qU langchain-openai

Instantiation

Now we can instantiate our model object and generate chat completions:

from langchain_core.messages import HumanMessage, SystemMessage
from langchain_core.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
)
from langchain_openai import ChatOpenAI

inference_server_url = "http://localhost:8000/v1"

llm = ChatOpenAI(
    model="mosaicml/mpt-7b",
    openai_api_key="EMPTY",
    openai_api_base=inference_server_url,
    max_tokens=5,
    temperature=0,
)

Invocation

messages = [
    SystemMessage(
        content="You are a helpful assistant that translates English to Italian."
    ),
    HumanMessage(
        content="Translate the following sentence from English to Italian: I love programming."
    ),
]
llm.invoke(messages)

AIMessage(content=' Io amo programmare', additional_kwargs={}, example=False)

Chaining

We can chain our model with a prompt template like so:

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate(
    [
        (
            "system",
            "You are a helpful assistant that translates {input_language} to {output_language}.",
        ),
        ("human", "{input}"),
    ]
)

chain = prompt | llm
chain.invoke(
    {
        "input_language": "English",
        "output_language": "German",
        "input": "I love programming.",
    }
)

API reference

For detailed documentation of all features and configurations exposed via langchain-openai, head to the API reference: python.langchain.com/api_reference/openai/chat_models/langchain_openai.chat_models.base.ChatOpenAI.html

Refer to the vLLM documentation as well.