evennia.contrib.rpg.llm.llm_client¶
LLM (Large Language Model) client, for communicating with an LLM backend. This can be used for generating texts for AI npcs, or for fine-tuning the LLM on a given prompt.
Note that running a LLM locally requires a lot of power, and ideally a powerful GPU. Testing this with CPU mode on a beefy laptop, still takes some 4s just on a very small model.
The server defaults to output suitable for a local server https://github.com/oobabooga/text-generation-webui, but could be used for other LLM servers too.
See the LLM instructions on that page for how to set up the server. You’ll also need a model file - there are thousands to try out on https://huggingface.co/models (you want Text Generation models specifically).
Optional Evennia settings (if not given, these defaults are used)¶
DEFAULT_LLM_HOST = “http://localhost:5000” DEFAULT_LLM_PATH = “/api/v1/generate” DEFAULT_LLM_HEADERS = {“Content-Type”: “application/json”} DEFAULT_LLM_PROMPT_KEYNAME = “prompt” DEFAULT_LLM_REQUEST_BODY = {…} # see below, this controls how to prompt the LLM server.
-
class
evennia.contrib.rpg.llm.llm_client.
StringProducer
(body)[source]¶ Bases:
object
Used for feeding a request body to the HTTP client.
-
class
evennia.contrib.rpg.llm.llm_client.
SimpleResponseReceiver
(status_code, d)[source]¶ Bases:
twisted.internet.protocol.Protocol
Used for pulling the response body out of an HTTP response.
-
dataReceived
(data)[source]¶ Called whenever data is received.
Use this method to translate to a higher-level message. Usually, some callback will be made upon the receipt of each complete protocol message.
- @param data: a string of indeterminate length. Please keep in mind
that you will probably need to buffer some data, as partial (or multiple) protocol messages may be received! I recommend that unit tests for protocols call through to this method with differing chunk sizes, down to one byte at a time.
-
connectionLost
(reason=<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>)[source]¶ Called when the connection is shut down.
Clear any circular references here, and any external references to this Protocol. The connection has been closed.
@type reason: L{twisted.python.failure.Failure}
-
-
class
evennia.contrib.rpg.llm.llm_client.
QuietHTTP11ClientFactory
(quiescentCallback, metadata)[source]¶ Bases:
twisted.web.client._HTTP11ClientFactory
Silences the obnoxious factory start/stop messages in the default client.
-
noisy
= False¶
-
-
class
evennia.contrib.rpg.llm.llm_client.
LLMClient
(on_bad_request=None)[source]¶ Bases:
object
A client for communicating with an LLM server.
-
__init__
(on_bad_request=None)[source]¶ Initialize self. See help(type(self)) for accurate signature.
-
get_response
(prompt)[source]¶ Get a response from the LLM server for the given npc.
- Parameters
prompt (str or list) – The prompt to send to the LLM server. If a list, this is assumed to be the chat history so far, and will be added to the prompt in a way suitable for the api.
- Returns
str –
- The generated text response. Will return an empty string
if there is an issue with the server, in which case the the caller is expected to handle this gracefully.
-