compress_pickle

dump(obj, path[, compression, ...])	Serialize an object and write it to a file-like object.
dumps(obj, compression[, pickler_method, ...])	Serialize an object and return its binary representation.
load(path[, compression, pickler_method, ...])	Load an object from its serialized binary representation stored in a file-like object.
loads(data, compression[, pickler_method, ...])	Load an object from its serialized binary representation.

A thin wrapper of standard pickle with standard compression libraries

compress_pickle.compress_pickle.dump(obj: Any, path: Union[str, bytes, os.PathLike, IO[bytes]], compression: Optional[str] = 'infer', pickler_method: str = 'pickle', pickler_kwargs: Optional[Dict[str, Any]] = None, mode: Optional[str] = None, *, set_default_extension: bool = True, **kwargs)

Serialize an object and write it to a file-like object.

It is possible to specify a desired serialization method and a compression protocol to use. For example, if gzip compression is specified, the object is serialized using the standard pickle.dump() and compressed using gzip.

Behind the scenes, this function only instantiates BaseCompresser and BasePicklerIO instances depending on the supplied names, and then calls compress_and_pickle() with them.

Parameters

• obj (Any) – The object that will be serialized

• path (Union[PathType, FileType]) – A path-like object (str, bytes, os.PathType) or a file-like object (io.BaseIO instances). The path to which to dump the obj.

• compression (Optional[str]) – The compression protocol to use. By default, the compression is inferred from the path’s extension. This compression name passed to get_compresser() to create a BaseCompresser instance. This instance will be used to create a file-like object onto which to write the serialized binary representation of obj. To see available compression protocols refer to get_known_compressions().

• pickler_method (str) – The name of the serialization method to use. This method name is passed to get_pickler() to create a BasePicklerIO instance. The default method is "pickle", which means that a BuiltinPicklerIO will be created by default, and its dump() will be used for serializing the obj. Refer to get_known_picklers() to see the available serialization methods.

• pickler_kwargs (Optional[Dict[str, Any]]) – An optional dictionary of keyword arguments to pass to dump() when obj is serialized. For example, this could be {"protocol": -1, "fix_imports": True} when the default "pickle" serialization method is used.

• mode (Optional[str]) – Mode with which to open the file buffer. The default changes according to the compression protocol. Refer to get_compression_write_mode() to see the defaults.

• set_default_extension (bool) – If True, the default extension given the provided compression protocol is set to the supplied path. Refer to get_default_compression_mapping() for the default extension registered to each compression method.

• kwargs – Any extra keyword arguments are passed to compress_pickle.utils.instantiate_compresser(), which in turn creates the compresser instance that will be used.

Notes

If the supplied path is a file-like object, dump does not close it before exiting. The users must handle the closing on their own. If the supplied path is a path-like object, dump opens and then closes the file after the writting process finishes.

compress_pickle.compress_pickle.dumps(obj: Any, compression: Optional[str], pickler_method: str = 'pickle', pickler_kwargs: Optional[Dict[str, Any]] = None, **kwargs) → bytes

Serialize an object and return its binary representation.

It is possible to specify a desired serialization method and a compression protocol to use. For example, if gzip compression is specified, the serialized object is compressed using gzip.

Behind the scenes, this function only instantiates BaseCompresser around a io.BinaryIO and BasePicklerIO instances depending on the supplied names, and then calls compress_and_pickle() with them. The contents of the io.BinaryIO are then returned.

Parameters

• obj (Any) – The object that will be serialized.

• compression (Optional[str]) – The compression protocol to use. By default, the compression is inferred from the path’s extension. This compression name passed to get_compresser() to create a BaseCompresser instance. This instance will be used to create a file-like object onto which to write the serialized binary representation of obj. To see available compression protocols refer to get_known_compressions().

• pickler_method (str) – The name of the serialization method to use. This method name is passed to get_pickler() to create a BasePicklerIO instance. The default method is "pickle", which means that a BuiltinPicklerIO will be created by default, and its dump() will be used for serializing the obj. Refer to get_known_picklers() to see the available serialization methods.

• pickler_kwargs (Optional[Dict[str, Any]]) – An optional dictionary of keyword arguments to pass to dump() when obj is serialized. For example, this could be {"protocol": -1, "fix_imports": True} when the default "pickle" serialization method is used.

Returns

The resulting bytes of the serialized obj after being compressed.

Return type

bytes

Notes

The compression argument is mandatory because it cannot be inferred.

compress_pickle.compress_pickle.load(path: Union[str, bytes, os.PathLike, IO[bytes]], compression: Optional[str] = 'infer', pickler_method: str = 'pickle', pickler_kwargs: Optional[Dict[str, Any]] = None, mode: Optional[str] = None, *, set_default_extension: bool = True, **kwargs) → Any

Load an object from its serialized binary representation stored in a file-like object.

It is possible to specify a desired serialization method and a compression protocol to use. For example, if gzip compression is specified, the contents of the supplied file-like object are uncompressed using gzip and then loaded using standard pickle.load().

Behind the scenes, this function only instantiates BaseCompresser and BasePicklerIO instances depending on the supplied names, and then calls uncompress_and_unpickle() with them.

Parameters

• path (Union[PathType, FileType]) – A path-like object (str, bytes, os.PathType) or a file-like object (io.BaseIO instances). The path to which to dump the obj.

• compression (Optional[str]) – The compression protocol to use. By default, the compression is inferred from the path’s extension. This compression name passed to get_compresser() to create a BaseCompresser instance. This instance will be used to open a file-like object from which to read the serialized binary representation of obj. To see available compression protocols refer to get_known_compressions().

• pickler_method (str) – The name of the serialization method to use. This method name is passed to get_pickler() to create a BasePicklerIO instance. The default method is "pickle", which means that a BuiltinPicklerIO will be created by default, and its load() will be used for unserializing the obj. Refer to get_known_picklers() to see the available serialization methods.

• pickler_kwargs (Optional[Dict[str, Any]]) – An optional dictionary of keyword arguments to pass to load() when the obj is loaded. For example, this could be {"fix_imports": True} when the default "pickle" serialization method is used.

• mode (Optional[str]) – Mode with which to open the file buffer. The default changes according to the compression protocol. Refer to get_compression_read_mode() to see the defaults.

• set_default_extension (bool) – If True, the default extension given the provided compression protocol is set to the supplied path. Refer to get_default_compression_mapping() for the default extension registered to each compression method.

• kwargs – Any extra keyword arguments are passed to compress_pickle.utils.instantiate_compresser(), which in turn creates the compresser instance that will be used.

Returns

obj – The object that is loaded from its compressed binary representation.

Return type

Any

Notes

If the supplied path is a file-like object, load does not close it before exiting. The users must handle the closing on their own. If the supplied path is a path-like object, load opens and then closes the file after the writting process finishes.

compress_pickle.compress_pickle.loads(data: bytes, compression: Optional[str], pickler_method: str = 'pickle', pickler_kwargs: Optional[Dict[str, Any]] = None, **kwargs) → Any

Load an object from its serialized binary representation.

It is possible to specify a desired serialization method and a compression protocol to use. For example, if gzip compression is specified, the supplied binary data is uncompressed with gzip and then loaded using standard pickle.load().

Behind the scenes, this function only instantiates BaseCompresser around a io.BinaryIO(data) stream and BasePicklerIO instances depending on the supplied names, and then calls uncompress_and_unpickle() with them.

Parameters

• data (bytes) – The bytes that contain the compressed serialized binary representation of the object that must be loaded.

• compression (Optional[str]) – The compression protocol to use. By default, the compression is inferred from the path’s extension. This compression name passed to get_compresser() to create a BaseCompresser instance. This instance will be used to open a file-like object from which to read the serialized binary representation of obj. To see available compression protocols refer to get_known_compressions().

• pickler_method (str) – The name of the serialization method to use. This method name is passed to get_pickler() to create a BasePicklerIO instance. The default method is "pickle", which means that a BuiltinPicklerIO will be created by default, and its load() will be used for unserializing the obj. Refer to get_known_picklers() to see the available serialization methods.

• pickler_kwargs (Optional[Dict[str, Any]]) – An optional dictionary of keyword arguments to pass to load() when the obj is loaded. For example, this could be {"fix_imports": True} when the default "pickle" serialization method is used.

• kwargs – Any extra keyword arguments are passed to compress_pickle.utils.instantiate_compresser(), which in turn creates the compresser instance that will be used.

Returns

obj – The object that is loaded from its compressed binary representation.

Return type

Any

Notes

The compression argument is mandatory because it cannot be inferred.

• Index

• Module Index

• Search Page