Skip to content
New: Star-import, signal unraveling, Python 3.12 support, and more

# pickling module

Utilities for pickling.

## rec_info_registry dict

Registry with instances of RecInfo keyed by RecInfo.id_.

Populate with the required information if any instance cannot be unpickled.

## dumps function

    dumps(
        obj,
        compression=None,
        compress_kwargs=None,
        **kwargs
    )

Pickle an object to a byte stream.

Uses `dill` when available.

For compression options, see `extensions`.

Keyword arguments `compress_kwargs` are passed to the `compress` method of the compressing package.

Other keyword arguments are passed to the dumps() method of the pickling package.

## get_class_from_id function

    get_class_from_id(
        class_id
    )

Get the class from a class id.

## get_compression_extensions function

    get_compression_extensions(
        cls_name=None
    )

Get all supported compression extensions from vectorbtpro._settings.

## get_id_from_class function

    get_id_from_class(
        obj
    )

Get the class id from a class.

If the object is an instance or a subclass of Pickleable and `Pickleable._rec_id` is not None, uses the reconstruction id. Otherwise, returns the path to the class definition with find_class().

## get_serialization_extensions function

    get_serialization_extensions(
        cls_name=None
    )

Get all supported serialization extensions from vectorbtpro._settings.

## load function

    load(
        path,
        compression=None,
        **kwargs
    )

Read a byte stream from a file and unpickle.

Can recognize the compression algorithm based on the extension (see dumps() for options).

Uses loads() for deserialization.

## loads function

    loads(
        bytes_,
        compression=None,
        **kwargs
    )

Unpickle an object from a byte stream.

Accepts the same options for `compression` as in the dumps() method.

Other keyword arguments are passed to the loads() method of the pickling package.

## reconstruct function

    reconstruct(
        cls,
        rec_state
    )

Reconstruct an instance using a class and a reconstruction state.

## save function

    save(
        obj,
        path=None,
        mkdir_kwargs=None,
        compression=None,
        **kwargs
    )

Pickle an object to a byte stream and write to a file.

Can recognize the compression algorithm based on the extension (see dumps() for options).

Uses dumps() for serialization.

## Pickleable class

    Pickleable()

Superclass that defines abstract properties and methods for pickle-able classes.

If any subclass cannot be pickled, override the Pickleable.rec_state property to return an instance of RecState to be used in reconstruction. If the class definition cannot be pickled (e.g., created dynamically), override its `_rec_id` with an arbitrary id string, dump/save the class, and before loading, map this id to the class in `rec_id_map`. This will use the mapped class to construct a new instance.

Subclasses

- Configured
- pdict

### decode_config class method

    Pickleable.decode_config(
        str_,
        parse_literals=True,
        run_code=True,
        pack_objects=True,
        use_refs=True,
        use_class_ids=True,
        code_context=None,
        parser_kwargs=None,
        check_type=True,
        **kwargs
    )

Decode an instance from a config string.

Can parse configs without sections. Sections can also become sub-dicts if their names use the dot notation. For example, section `a.b` will become a sub-dict of the section `a` and section `a.b.c` will become a sub-dict of the section `a.b`. You don't have to define the section `a` explicitly, it will automatically become the outermost key.

If a section contains only one pair `_ = _`, it will become an empty dict.

If `parse_literals` is True, will detect any Python literals and containers such as `True` and `[]`. Will also understand `np.nan`, `np.inf`, and `-np.inf`.

If `run_code` is True, will run any Python code prepended with `!`. Will use the context `code_context` together with already defined `np` (NumPy), `pd` (Pandas), and `vbt` (vectorbtpro).

Warning

Unpickling byte streams and running code has important security implications. Don't attempt to parse configs coming from untrusted sources as those can contain malicious code!

If `pack_objects` is True, will look for class paths prepended with `@` in section names, construct an instance of RecState (any other keyword arguments will be included to `init_kwargs`), and finally use reconstruct() to reconstruct the unpacked object.

If `use_refs` is True, will substitute references prepended with `&` for actual objects. Constructs a DAG using graphlib.

If `use_class_ids` is True, will substitute any class ids prepended with `@` with the corresponding class.

Other keyword arguments are forwarded to Pickleable.decode_config_node().

Usage

- File `types.ini`:

    string = 'hello world'
    boolean = False
    int = 123
    float = 123.45
    exp_float = 1e-10
    nan = np.nan
    inf = np.inf
    numpy = !np.array([1, 2, 3])
    pandas = !pd.Series([1, 2, 3])
    expression = !dict(sub_dict2=dict(some="value"))
    mult_expression = !import math; math.floor(1.5)

    >>> from vectorbtpro import *

    >>> vbt.pprint(vbt.pdict.load("types.ini"))
    pdict(
        string='hello world',
        boolean=False,
        int=123,
        float=123.45,
        exp_float=1e-10,
        nan=np.nan,
        inf=np.inf,
        numpy=<numpy.ndarray object at 0x7fe1bf84f690 of shape (3,)>,
        pandas=<pandas.core.series.Series object at 0x7fe1c9a997f0 of shape (3,)>,
        expression=dict(
            sub_dict2=dict(
                some='value'
            )
        ),
        mult_expression=1
    )

- File `refs.ini`:

    [top]
    sr = &top.sr

    [top.sr @pandas.Series]
    data = [10756.12, 10876.76, 11764.33]
    index = &top.sr.index
    name = 'Open time'

    [top.sr.index @pandas.DatetimeIndex]
    data = ["2023-01-01", "2023-01-02", "2023-01-03"]

    >>> vbt.pdict.load("refs.ini")["sr"]
    2023-01-01    10756.12
    2023-01-02    10876.76
    2023-01-03    11764.33
    Name: Open time, dtype: float64

### decode_config_node class method

    Pickleable.decode_config_node(
        key,
        value,
        **kwargs
    )

Decode a config node.

### dumps method

    Pickleable.dumps(
        rec_state_only=False,
        **kwargs
    )

Pickle the instance to a byte stream.

Optionally, you can set `rec_state_only` to True if the instance will be later unpickled directly by the class.

### encode_config method

    Pickleable.encode_config(
        top_name=None,
        unpack_objects=True,
        compress_unpacked=True,
        use_refs=True,
        use_class_ids=True,
        nested=True,
        to_dict=False,
        parser_kwargs=None,
        **kwargs
    )

Encode the instance to a config string.

Based on Pickleable.rec_state. Raises an error if None.

Encodes to a format that is guaranteed to be parsed using Pickleable.decode_config(). Otherwise, an error will be thrown. If any object cannot be represented using a string, uses dumps() to convert it to a byte stream.

When `unpack_objects` is True and an object is an instance of Pickleable, saves its reconstruction state to a separate section rather than the byte stream. Appends `@` and class name to the section name. If `compress_unpacked` is True, will hide keys in RecState that have empty values. Keys in RecState will be appended with `~` to avoid collision with user-defined keys having the same name.

If `use_refs` is True, out of unhashable objects sharing the same id, only the first one will be defined while others will store the reference (`&` + key path) to the first one.

Note

The initial order of keys can be preserved only by using references.

If `use_class_ids` is True, substitutes any class defined as a value by its id instead of pickling its definition. If get_id_from_class() returns None, will pickle the definition.

If the instance is nested, set `nested` to True to represent each sub-dict as a section.

Other keyword arguments are forwarded to Pickleable.encode_config_node().

### encode_config_node method

    Pickleable.encode_config_node(
        key,
        value,
        **kwargs
    )

Encode a config node.

### file_exists class method

    Pickleable.file_exists(
        *args,
        **kwargs
    )

Return whether a file already exists.

Resolves the file path using Pickleable.resolve_file_path().

### getsize method

    Pickleable.getsize(
        readable=True,
        **kwargs
    )

Get size of this object.

### load class method

    Pickleable.load(
        path=None,
        file_format=None,
        compression=None,
        **kwargs
    )

Unpickle/decode the instance from a file.

Resolves the file path using Pickleable.resolve_file_path().

### loads class method

    Pickleable.loads(
        bytes_,
        check_type=True,
        **kwargs
    )

Unpickle an instance from a byte stream.

### modify_state class method

    Pickleable.modify_state(
        rec_state
    )

Modify the reconstruction state before reconstruction.

### rec_state property

Reconstruction state of the type RecState.

### resolve_file_path class method

    Pickleable.resolve_file_path(
        path=None,
        file_format=None,
        compression=None,
        for_save=False
    )

Resolve a file path.

A file must have either one or two extensions: file format (required) and compression (optional). For file format options, see `pickle_extensions` and `config_extensions`. For compression options, see `compression_extensions`. Each can be provided either via a suffix in `path`, or via the argument `file_format` and `compression` respectively.

When saving, uses `file_format` and `compression` from pickling as default options. When loading, searches for matching files in the current directory.

Path can be also None or directory, in such a case the file name will be set to the class name.

### save method

    Pickleable.save(
        path=None,
        file_format=None,
        compression=None,
        mkdir_kwargs=None,
        **kwargs
    )

Pickle/encode the instance and save to a file.

Resolves the file path using Pickleable.resolve_file_path().

## RecInfo class

    RecInfo(
        id_,
        cls,
        modify_state=None
    )

Class that represents information needed to reconstruct an instance.

Method generated by attrs for class RecInfo.

### cls class variable

Class.

### id_ class variable

Identifier.

### modify_state class variable

Callback to modify the reconstruction state.

### register method

    RecInfo.register()

Register self in rec_info_registry.

## RecState class

    RecState(
        init_args=NOTHING,
        init_kwargs=NOTHING,
        attr_dct=NOTHING
    )

Class that represents a state used to reconstruct an instance.

Method generated by attrs for class RecState.

### attr_dct class variable

Dictionary with names and values of writeable attributes.

### init_args class variable

Positional arguments used in initialization.

### init_kwargs class variable

Keyword arguments used in initialization.

## pdict class

    pdict(
        *args,
        **kwargs
    )

Pickleable dict.

Superclasses

- Comparable
- Pickleable
- Prettified
- `builtins.dict`

Inherited members

- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.rec_state
- Pickleable.resolve_file_path()
- Pickleable.save()
- Prettified.prettify()

Subclasses

- Config
- atomic_dict
- child_dict
- index_dict
- key_dict
- pfopt_func_dict
- run_arg_dict
- run_func_dict

### equals method

    pdict.equals(
        other,
        check_types=True,
        **kwargs
    )

Check two objects for equality.

### load_update method

    pdict.load_update(
        path=None,
        clear=False,
        **kwargs
    )

Load dumps from a file and update this instance in-place.

#### Cookie consent

We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better.

- Google Analytics
- GitHub

Manage settings