evennia.utils.dbserialize

This module handles serialization of arbitrary python structural data, intended primarily to be stored in the database. It also supports storing Django model instances (which plain pickle cannot do).

This serialization is used internally by the server, notably for storing data in Attributes and for piping data to process pools.

The purpose of dbserialize is to handle all forms of data. For well-structured non-arbitrary exchange, such as communicating with a rich web client, a simpler JSON serialization makes more sense.

This module also implements the SaverList, SaverDict and SaverSet classes. These are iterables that track their position in a nested structure and makes sure to send updates up to their root. This is used by Attributes - without it, one would not be able to update mutables in-situ, e.g obj.db.mynestedlist[3][5] = 3 would never be saved and be out of sync with the database.

evennia.utils.dbserialize.to_pickle(data)

This prepares data on arbitrary form to be pickled. It handles any nested structure and returns data on a form that is safe to pickle (including having converted any database models to their internal representation). We also convert any Saver*-type objects back to their normal representations, they are not pickle-safe.

Parameters:

data (any) – Data to pickle.

Returns:

data (any) – Pickled data.

evennia.utils.dbserialize.from_pickle(data, db_obj=None)

This should be fed a just de-pickled data object. It will be converted back to a form that may contain database objects again. Note that if a database object was removed (or changed in-place) in the database, None will be returned.

Parameters:

• data (any) – Pickled data to unpickle.

• db_obj (Atribute, any) – This is the model instance (normally an Attribute) that _Saver*-type iterables (_SaverList etc) will save to when they update. It must have a ‘value’ property that saves assigned data to the database. Skip if not serializing onto a given object. If db_obj is given, this function will convert lists, dicts and sets to their _SaverList, _SaverDict and _SaverSet counterparts.

Returns:

data (any) – Unpickled data.

evennia.utils.dbserialize.do_pickle(data)

Perform pickle to string

evennia.utils.dbserialize.do_unpickle(data)

Retrieve pickle from pickled string

evennia.utils.dbserialize.dbserialize(data)

Serialize to pickled form in one step

evennia.utils.dbserialize.dbunserialize(data, db_obj=None)

Un-serialize in one step. See from_pickle for help db_obj.

class evennia.utils.dbserialize.ContentType(id, app_label, model)

Bases: Model

exception DoesNotExist

Bases: ObjectDoesNotExist

exception MultipleObjectsReturned

Bases: MultipleObjectsReturned

app_label

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

property app_labeled_name

get_all_objects_for_this_type(**kwargs)

Return all objects of this type for the keyword arguments given.

get_object_for_this_type(using=None, **kwargs)

Return an object of this type for the keyword arguments given. Basically, this is a proxy around this object_type’s get_object() model method. The ObjectNotExist exception, if thrown, will not be caught, so code that calls this method should catch it.

id

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

logentry_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

**Parent.children** is a **ReverseManyToOneDescriptor** instance.

Most of the implementation is delegated to a dynamically defined manager class built by **create_forward_many_to_many_manager()** defined below.

model

A wrapper for a deferred-loading field. When the value is read from this object the first time, the query is executed.

model_class()

Return the model class for this type of content.

property name

natural_key()

objects = <django.contrib.contenttypes.models.ContentTypeManager object>

permission_set

Accessor to the related objects manager on the reverse side of a many-to-one relation.

In the example:

class Child(Model):
    parent = ForeignKey(Parent, related_name='children')

**Parent.children** is a **ReverseManyToOneDescriptor** instance.

Most of the implementation is delegated to a dynamically defined manager class built by **create_forward_many_to_many_manager()** defined below.

class evennia.utils.dbserialize.IntFlag(new_class_name, /, names, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: int, ReprEnum, Flag

Support for integer-based Flags

class evennia.utils.dbserialize.MutableMapping

Bases: Mapping

A MutableMapping is a generic container for associating key/value pairs.

This class provides concrete generic implementations of all methods except for __getitem__, __setitem__, __delitem__, __iter__, and __len__.

clear() → None.  Remove all items from D.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() → (k, v), remove and return some (key, value) pair

as a 2-tuple; but raise KeyError if D is empty.

setdefault(k[, d]) → D.get(k,d), also set D[k]=d if k not in D

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E present and has a .keys() method, does: for k in E.keys(): D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

class evennia.utils.dbserialize.MutableSequence

Bases: Sequence

All the operations on a read-write sequence.

Concrete subclasses must provide __new__ or __init__, __getitem__, __setitem__, __delitem__, __len__, and insert().

append(value)

S.append(value) – append value to the end of the sequence

clear() → None -- remove all items from S

extend(values)

S.extend(iterable) – extend sequence by appending elements from the iterable

abstractmethod insert(index, value)

S.insert(index, value) – insert value before index

pop([index]) → item -- remove and return item at index (default last).

Raise IndexError if list is empty or index is out of range.

remove(value)

S.remove(value) – remove first occurrence of value. Raise ValueError if the value is not present.

reverse()

S.reverse() – reverse IN PLACE

class evennia.utils.dbserialize.MutableSet

Bases: Set

A mutable set is a finite, iterable container.

This class provides concrete generic implementations of all methods except for __contains__, __iter__, __len__, add(), and discard().

To override the comparisons (presumably for speed, as the semantics are fixed), all you have to do is redefine __le__ and then the other operations will automatically follow suit.

abstractmethod add(value)

Add an element.

clear()

This is slow (creates N new iterators!) but effective.

abstractmethod discard(value)

Remove an element. Do not raise an exception if absent.

pop()

Return the popped value. Raise KeyError if empty.

remove(value)

Remove an element. If not a member, raise a KeyError.

exception evennia.utils.dbserialize.ObjectDoesNotExist

Bases: Exception

The requested object does not exist

silent_variable_failure = True

class evennia.utils.dbserialize.OrderedDict

Bases: dict

Dictionary that remembers insertion order

__init__(*args, **kwargs)

clear() → None.  Remove all items from od.

copy() → a shallow copy of od

classmethod fromkeys(iterable, value=None)

Create a new ordered dictionary with keys from iterable and values set to value.

items()

Return a set-like object providing a view on the dict’s items.

keys()

Return a set-like object providing a view on the dict’s keys.

move_to_end(key, last=True)

Move an existing element to the end (or beginning if last is false).

Raise KeyError if the element does not exist.

pop(key[, default]) → v, remove specified key and return the corresponding value.

If the key is not found, return the default if given; otherwise, raise a KeyError.

popitem(last=True)

Remove and return a (key, value) pair from the dictionary.

Pairs are returned in LIFO order if last is true or FIFO order if false.

setdefault(key, default=None)

Insert key with a value of default if key is not in the dictionary.

Return the value for key if key is in the dictionary, else default.

update([E, ]**F) → None.  Update D from mapping/iterable E and F.

If E is present and has a .keys() method, then does: for k in E.keys(): D[k] = E[k] If E is present and lacks a .keys() method, then does: for k, v in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

values()

Return an object providing a view on the dict’s values.

class evennia.utils.dbserialize.SafeString

Bases: str, SafeData

A str subclass that has been specifically marked as “safe” for HTML output purposes.

exception evennia.utils.dbserialize.UnpicklingError

Bases: PickleError

class evennia.utils.dbserialize.defaultdict

Bases: dict

defaultdict(default_factory=None, /, […]) –> dict with default factory

The default factory is called without arguments to produce a new value when a key is not present, in __getitem__ only. A defaultdict compares equal to a dict with the same items. All remaining arguments are treated the same as if they were passed to the dict constructor, including keyword arguments.

__init__(*args, **kwargs)

copy() → a shallow copy of D.

default_factory

Factory for default value called by __missing__().

class evennia.utils.dbserialize.deque

Bases: object

A list-like sequence optimized for data accesses near its endpoints.

__init__(*args, **kwargs)

append(item, /)

Add an element to the right side of the deque.

appendleft(item, /)

Add an element to the left side of the deque.

clear()

Remove all elements from the deque.

copy()

Return a shallow copy of a deque.

count(value, /)

Return number of occurrences of value.

extend(iterable, /)

Extend the right side of the deque with elements from the iterable.

extendleft(iterable, /)

Extend the left side of the deque with elements from the iterable.

index()

Return first index of value.

Raises ValueError if the value is not present.

insert(index, value, /)

Insert value before index.

maxlen

maximum size of a deque or None if unbounded

pop()

Remove and return the rightmost element.

popleft()

Remove and return the leftmost element.

remove(value, /)

Remove first occurrence of value.

reverse()

Reverse IN PLACE.

rotate(n=1, /)

Rotate the deque n steps to the right. If n is negative, rotates left.

evennia.utils.dbserialize.deserialize(obj)

Make sure to fully decouple a structure from the database, by turning all _Saver*-mutables inside it back into their normal Python forms.

evennia.utils.dbserialize.dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None)

Return the pickled representation of the object as a bytes object.

The optional protocol argument tells the pickler to use the given protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default protocol is 4. It was introduced in Python 3.4, and is incompatible with previous versions.

Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.

If fix_imports is True and protocol is less than 3, pickle will try to map the new Python 3 names to the old module names used in Python 2, so that the pickle data stream is readable with Python 2.

If buffer_callback is None (the default), buffer views are serialized into file as part of the pickle stream. It is an error if buffer_callback is not None and protocol is None or smaller than 5.

evennia.utils.dbserialize.is_iter(obj)

Checks if an object behaves iterably.

Parameters:

obj (any) – Entity to check for iterability.

Returns:

is_iterable (bool) – If obj is iterable or not.

Notes

Strings are not accepted as iterable (although they are actually iterable), since string iterations are usually not what we want to do with a string.

evennia.utils.dbserialize.loads(data, /, *, fix_imports=True, encoding='ASCII', errors='strict', buffers=())

Read and return an object from the given pickle data.

The protocol version of the pickle is detected automatically, so no protocol argument is needed. Bytes past the pickled object’s representation are ignored.

Optional keyword arguments are fix_imports, encoding and errors, which are used to control compatibility support for pickle stream generated by Python 2. If fix_imports is True, pickle will try to map the old Python 2 names to the new names used in Python 3. The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2; these default to ‘ASCII’ and ‘strict’, respectively. The encoding can be ‘bytes’ to read these 8-bit string instances as bytes objects.

evennia.utils.dbserialize.pack_dbobj(item)

Check and convert django database objects to an internal representation.

Parameters:

item (any) – A database entity to pack

Returns:

packed (any or tuple) –

Either returns the original input item

or the packing tuple (“__packed_dbobj__”, key, creation_time, id).

evennia.utils.dbserialize.pack_session(item)

Handle the safe serializion of Sessions objects (these contain hidden references to database objects (accounts, puppets) so they can’t be safely serialized).

Parameters:

item (Session)) – This item must have all properties of a session before entering this call.

Returns:

packed (tuple or None) –

A session-packed tuple on the form

(__packed_session__, sessid, conn_time). If this sessid does not match a session in the Session handler, None is returned.

evennia.utils.dbserialize.to_bytes(text, session=None)

Try to encode the given text to bytes, using encodings from settings or from Session. Will always return a bytes, even if given something that is not str or bytes.

Parameters:

• text (any) – The text to encode to bytes. If bytes, return unchanged. If not a str, convert to str before converting.

• session (Session, optional) – A Session to get encoding info from. Will try this before falling back to settings.ENCODINGS.

Returns:

encoded_text (bytes) –

the encoded text following the session’s protocol flag followed by the

encodings specified in settings.ENCODINGS. If all attempt fail, log the error and send the text with “?” in place of problematic characters. If the specified encoding cannot be found, the protocol flag is reset to utf-8. In any case, returns bytes.

Notes

If text is already bytes, return it as is.

evennia.utils.dbserialize.unpack_dbobj(item)

Check and convert internal representations back to Django database models.

Parameters:

item (packed_dbobj) – The fact that item is a packed dbobj should be checked before this call.

Returns:

unpacked (any) –

Either the original input or converts the

internal store back to a database representation (its typeclass is returned if applicable).

evennia.utils.dbserialize.unpack_session(item)

Check and convert internal representations back to Sessions.

Parameters:

item (packed_session) – The fact that item is a packed session should be checked before this call.

Returns:

unpacked (any) –

Either the original input or converts the

internal store back to a Session. If Session no longer exists, None will be returned.

evennia.utils.dbserialize.update_wrapper(wrapper, wrapped, assigned=('__module__', '__name__', '__qualname__', '__doc__', '__annotations__', '__type_params__'), updated=('__dict__',))

Update a wrapper function to look like the wrapped function

wrapper is the function to be updated wrapped is the original function assigned is a tuple naming the attributes assigned directly from the wrapped function to the wrapper function (defaults to functools.WRAPPER_ASSIGNMENTS) updated is a tuple naming the attributes of the wrapper that are updated with the corresponding attribute from the wrapped function (defaults to functools.WRAPPER_UPDATES)

evennia.utils.dbserialize.uses_database(name='sqlite3')

Checks if the game is currently using a given database. This is a shortcut to having to use the full backend name.

Parameters:

name (str) – One of ‘sqlite3’, ‘mysql’, ‘postgresql’ or ‘oracle’.

Returns:

uses (bool) – If the given database is used or not.