io

compress_pickle is intended to be easily extensible. This means that it should be easy to add new compresser and picklerIO classes and customize the functionality of serializing and unserializing. For this reason, the two main core functions: compress_pickle.io.base.compress_and_pickle() and compress_pickle.io.base.uncompress_and_unpickle() are implemented using functools.singledispatch(). compress_pickle provides a base implementation of both functions that work with compress_pickle.compressers.base.BaseCompresser, but you can easily register a custom way of handling a specific compresser by calling

compress_and_pickle.register(SomeClass)
def my_custom_implementation(compresser: SomeClass, pickler: BasePicklerIO, obj: Any, **kwargs):
    # do something special

Combining this, with the compresser and picklerIO registry capabilities, you will be able to create new custom compressers and serializers, register them and then use them with simple calls to dump() and load().

base.compress_and_pickle(compresser, ...)

Take an object, serialize it and write it to a compresser's stream.

base.uncompress_and_unpickle(compresser, ...)

Load and uncompress an object from a compresser's stream.
compress_pickle.io.base.compress_and_pickle(compresser: Any, pickler: compress_pickle.picklers.base.BasePicklerIO, obj: Any, **kwargs)[source]

Take an object, serialize it and write it to a compresser’s stream.

This is the main serialization function. It works as a singledispatch function, and its implementation is dependent on the type of the supplied compresser. The default implementation only supports BaseCompresser instances.

The compresser is used to get the stream onto which to write the serialized obj. The pickler instance is the object that is responsible for dumping the obj to the compresser’s stream.

Parameters

  • compresser (BaseCompresser) – The compresser instance that provides the file-like object onto which the obj must be written.

  • pickler (BasePicklerIO) – The object that is responsible for serializing the obj and writting its binary representation into the file-like stream provided by the compresser.

  • obj (Any) – The object that you wish to serialize, compress and write.

  • **kwargs – Any extra keyword arguments are passed to the pickler’s dump method.

Raises

NotImplementedError – If the compresser is not an instance of BaseCompresser

compress_pickle.io.base.uncompress_and_unpickle(compresser: Any, pickler: compress_pickle.picklers.base.BasePicklerIO, **kwargs) → Any[source]

Load and uncompress an object from a compresser’s stream.

This is the main loading function. It works as a singledispatch function, and its implementation is dependent on the type of the supplied compresser. The default implementation only supports BaseCompresser instances.

The compresser is used to get the stream from which to load the serialized obj. The pickler instance is the object that is responsible for loading the obj from the compresser’s stream.

Parameters

  • compresser (BaseCompresser) – The compresser instance that provides the file-like object from which to load the obj.

  • pickler (BasePicklerIO) – The object that is responsible for reading the contents of the file-like stream taken from the compresser and loading the serialized object

  • **kwargs – Any extra keyword arguments are passed to the pickler’s load method.

Returns

The resulting uncompressed and unserialized object.

Return type

Any

Raises

NotImplementedError – If the compresser is not an instance of BaseCompresser