evennia.contrib.rpg.llm.llm_client

LLM (Large Language Model) client, for communicating with an LLM backend. This can be used for generating texts for AI npcs, or for fine-tuning the LLM on a given prompt.

Note that running a LLM locally requires a lot of power, and ideally a powerful GPU. Testing this with CPU mode on a beefy laptop, still takes some 4s just on a very small model.

The server defaults to output suitable for a local server https://github.com/oobabooga/text-generation-webui, but could be used for other LLM servers too.

See the LLM instructions on that page for how to set up the server. You’ll also need a model file - there are thousands to try out on https://huggingface.co/models (you want Text Generation models specifically).

Optional Evennia settings (if not given, these defaults are used)

DEFAULT_LLM_HOST = “http://localhost:5000” DEFAULT_LLM_PATH = “/api/v1/generate” DEFAULT_LLM_HEADERS = {“Content-Type”: “application/json”} DEFAULT_LLM_PROMPT_KEYNAME = “prompt” DEFAULT_LLM_REQUEST_BODY = {…} # see below, this controls how to prompt the LLM server.

class evennia.contrib.rpg.llm.llm_client.StringProducer(body)

Bases: object

Used for feeding a request body to the HTTP client.

__init__(body)

startProducing(consumer)

pauseProducing()

stopProducing()

class evennia.contrib.rpg.llm.llm_client.SimpleResponseReceiver(status_code, d)

Bases: Protocol

Used for pulling the response body out of an HTTP response.

__init__(status_code, d)

dataReceived(data)

Called whenever data is received.

Use this method to translate to a higher-level message. Usually, some callback will be made upon the receipt of each complete protocol message.

@param data: a string of indeterminate length. Please keep in mind

that you will probably need to buffer some data, as partial (or multiple) protocol messages may be received! I recommend that unit tests for protocols call through to this method with differing chunk sizes, down to one byte at a time.

connectionLost(reason=<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>)

Called when the connection is shut down.

Clear any circular references here, and any external references to this Protocol. The connection has been closed.

@type reason: L{twisted.python.failure.Failure}

class evennia.contrib.rpg.llm.llm_client.QuietHTTP11ClientFactory(quiescentCallback, metadata)

Bases: _HTTP11ClientFactory

Silences the obnoxious factory start/stop messages in the default client.

noisy = False

class evennia.contrib.rpg.llm.llm_client.LLMClient(on_bad_request=None)

Bases: object

A client for communicating with an LLM server.

__init__(on_bad_request=None)

get_response(prompt)

Get a response from the LLM server for the given npc.

Parameters:

prompt (str or list) – The prompt to send to the LLM server. If a list, this is assumed to be the chat history so far, and will be added to the prompt in a way suitable for the api.

Returns:

str –

The generated text response. Will return an empty string

if there is an issue with the server, in which case the the caller is expected to handle this gracefully.