KoboldAI API

KoboldAI is a “a browser-based front-end for AI-assisted writing with multiple local & remote AI models…”. It has a public and local API that is able to be used in langchain.

This example goes over how to use LangChain with that API.

Documentation can be found in the browser adding /api to the end of your endpoint (i.e 127.0.0.1/:5000/api).

from langchain_community.llms import KoboldApiLLM

Replace the endpoint seen below with the one shown in the output after starting the webui with —api or —public-api

Optionally, you can pass in parameters like temperature or max_length

llm = KoboldApiLLM(endpoint="http://192.168.1.144:5000", max_length=80)

response = llm.invoke(
    "### Instruction:\nWhat is the first book of the bible?\n### Response:"
)