Skip to content
New: Star-import, signal unraveling, Python 3.12 support, and more

# execution module

Engines for executing functions.

## build_serial_chunk function

    build_serial_chunk(
        funcs_args
    )

Build a serial chunk.

## execute function

    execute(
        funcs_args,
        engine='serial',
        n_calls=None,
        min_size=None,
        n_chunks=None,
        chunk_len=None,
        chunk_meta=None,
        distribute=None,
        warmup=None,
        in_chunk_order=False,
        pre_execute_func=None,
        pre_execute_kwargs=None,
        pre_chunk_func=None,
        pre_chunk_kwargs=None,
        post_chunk_func=None,
        post_chunk_kwargs=None,
        post_execute_func=None,
        post_execute_kwargs=None,
        post_execute_on_sorted=False,
        show_progress=None,
        progress_desc=None,
        pbar_kwargs=None,
        template_context=None,
        engine_kwargs=None,
        **kwargs
    )

Execute using an engine.

Supported values for `engine`:

- Name of the engine (see supported engines)
- Subclass of ExecutionEngine - initializes with `kwargs` and `engine_kwargs`
- Instance of ExecutionEngine - calls ExecutionEngine.execute() with `n_calls`
- Callable - passes `funcs_args`, `n_calls` (if not None), and `kwargs` and `engine_kwargs`

Can execute per chunk if `chunk_meta` is provided. Otherwise, if any of `n_chunks` and `chunk_len` are set, passes them to yield_chunk_meta() to generate `chunk_meta`. Arguments `n_chunks` and `chunk_len` can be set globally in the engine-specific settings. Set `n_chunks` and `chunk_len` to 'auto' to set them to the number of cores.

If `distribute` is "calls", distributes calls within each chunk. If indices in `chunk_meta` are perfectly sorted and `funcs_args` is an iterable, iterates over `funcs_args` to avoid converting it into a list. Otherwise, iterates over `chunk_meta`. If `in_chunk_order` is True, returns the outputs in the order they appear in `chunk_meta`. Otherwise, always returns them in the same order as in `funcs_args`.

If `distribute` is "chunks", distributes chunks. For this, executes calls within each chunk serially using execute_serially(). Also, compresses each chunk such that each unique function, positional argument, and keyword argument is serialized only once.

If `funcs_args` is a custom template, substitutes it once `chunk_meta` is established. Use `template_context` as an additional context. All the resolved functions and arguments will be immediately passed to the executor.

If `pre_chunk_func` is not None, calls the function before processing a chunk. If it returns anything other than None, the returned object will be appended to the outputs and the chunk won't be executed. This enables use cases such as caching. If `post_chunk_func` is not None, calls the function after processing the chunk. It should return either None to keep the old call outputs, or return new ones. Will also substitute any templates in `pre_chunk_kwargs` and `post_chunk_kwargs` and pass them as keyword arguments. The following additional arguments are available in the contexts: the index of the current chunk `chunk_idx`, the list of call indices `call_indices` in the chunk, the list of call outputs `call_outputs` returned by executing the chunk (only for `post_chunk_func`), and whether the chunk was executed `chunk_executed` or otherwise returned by `pre_chunk_func` (only for `post_chunk_func`).

Note

The both callbacks above are effective only when `distribute` is "calls" and chunking is enabled.

If `pre_execute_func` is not None, calls the function before processing all calls. Should return nothing (None). Will also substitute any templates in `post_execute_kwargs` and pass them as keyword arguments. The following additional arguments are available in the context: the number of chunks `n_chunks`.

If `post_execute_func` is not None, calls the function after processing all calls. Will also substitute any templates in `post_execute_kwargs` and pass them as keyword arguments. Should return either None to keep the default outputs or return the new ones. The following additional arguments are available in the context: the number of chunks `n_chunks` and the generated flattened list of outputs `outputs`. If `post_execute_on_sorted` is True, will run the callback after sorting the call indices.

Info

Chunks are processed sequentially, while functions within each chunk can be processed distributively.

Supported engines can be found in `engines` in execution.

## execute_serially function

    execute_serially(
        funcs_args,
        id_objs
    )

Execute serially.

## pass_kwargs_as_args function

    pass_kwargs_as_args(
        func,
        args,
        kwargs
    )

Helper function for `pathos.pools.ParallelPool`.

## DaskEngine class

    DaskEngine(
        compute_kwargs=None,
        **kwargs
    )

Class for executing functions in parallel using Dask.

For defaults, see `engines.dask` in execution.

Note

Use multi-threading mainly on numeric code that releases the GIL (like NumPy, Pandas, Scikit-Learn, Numba).

Superclasses

- Cacheable
- Chainable
- Comparable
- Configured
- ExecutionEngine
- HasSettings
- Pickleable
- Prettified

Inherited members

- Cacheable.get_ca_setup()
- Chainable.pipe()
- Configured.copy()
- Configured.equals()
- Configured.get_writeable_attrs()
- Configured.prettify()
- Configured.replace()
- Configured.resolve_merge_kwargs()
- Configured.update_config()
- ExecutionEngine.config
- ExecutionEngine.execute()
- ExecutionEngine.rec_state
- HasSettings.get_path_setting()
- HasSettings.get_path_settings()
- HasSettings.get_setting()
- HasSettings.get_settings()
- HasSettings.has_path_setting()
- HasSettings.has_path_settings()
- HasSettings.has_setting()
- HasSettings.has_settings()
- HasSettings.reset_settings()
- HasSettings.resolve_setting()
- HasSettings.resolve_settings_paths()
- HasSettings.set_settings()
- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.resolve_file_path()
- Pickleable.save()

### compute_kwargs property

Keyword arguments passed to `dask.compute`.

## ExecutionEngine class

    ExecutionEngine(
        **config
    )

Abstract class for executing functions.

Superclasses

- Cacheable
- Chainable
- Comparable
- Configured
- HasSettings
- Pickleable
- Prettified

Inherited members

- Cacheable.get_ca_setup()
- Chainable.pipe()
- Configured.config
- Configured.copy()
- Configured.equals()
- Configured.get_writeable_attrs()
- Configured.prettify()
- Configured.rec_state
- Configured.replace()
- Configured.resolve_merge_kwargs()
- Configured.update_config()
- HasSettings.get_path_setting()
- HasSettings.get_path_settings()
- HasSettings.get_setting()
- HasSettings.get_settings()
- HasSettings.has_path_setting()
- HasSettings.has_path_settings()
- HasSettings.has_setting()
- HasSettings.has_settings()
- HasSettings.reset_settings()
- HasSettings.resolve_setting()
- HasSettings.resolve_settings_paths()
- HasSettings.set_settings()
- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.resolve_file_path()
- Pickleable.save()

Subclasses

- DaskEngine
- MpireEngine
- PathosEngine
- ProcessPoolEngine
- RayEngine
- SerialEngine
- ThreadPoolEngine

### execute method

    ExecutionEngine.execute(
        funcs_args,
        n_calls=None
    )

Run an iterable of tuples out of a function, arguments, and keyword arguments.

Provide `n_calls` in case `funcs_args` is a generator and the underlying engine needs it.

## MpireEngine class

    MpireEngine(
        init_kwargs=None,
        apply_kwargs=None,
        **kwargs
    )

Class for executing functions using `WorkerPool` from `mpire`.

For defaults, see `engines.mpire` in execution.

Superclasses

- Cacheable
- Chainable
- Comparable
- Configured
- ExecutionEngine
- HasSettings
- Pickleable
- Prettified

Inherited members

- Cacheable.get_ca_setup()
- Chainable.pipe()
- Configured.copy()
- Configured.equals()
- Configured.get_writeable_attrs()
- Configured.prettify()
- Configured.replace()
- Configured.resolve_merge_kwargs()
- Configured.update_config()
- ExecutionEngine.config
- ExecutionEngine.execute()
- ExecutionEngine.rec_state
- HasSettings.get_path_setting()
- HasSettings.get_path_settings()
- HasSettings.get_setting()
- HasSettings.get_settings()
- HasSettings.has_path_setting()
- HasSettings.has_path_settings()
- HasSettings.has_setting()
- HasSettings.has_settings()
- HasSettings.reset_settings()
- HasSettings.resolve_setting()
- HasSettings.resolve_settings_paths()
- HasSettings.set_settings()
- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.resolve_file_path()
- Pickleable.save()

### apply_kwargs property

Keyword arguments passed to `WorkerPool.async_apply`.

### init_kwargs property

Keyword arguments used to initialize `WorkerPool`.

## PathosEngine class

    PathosEngine(
        pool_type=None,
        init_kwargs=None,
        timeout=None,
        sleep=None,
        show_progress=None,
        pbar_kwargs=None,
        join_pool=None,
        **kwargs
    )

Class for executing functions using `pathos`.

For defaults, see `engines.pathos` in execution.

Superclasses

- Cacheable
- Chainable
- Comparable
- Configured
- ExecutionEngine
- HasSettings
- Pickleable
- Prettified

Inherited members

- Cacheable.get_ca_setup()
- Chainable.pipe()
- Configured.copy()
- Configured.equals()
- Configured.get_writeable_attrs()
- Configured.prettify()
- Configured.replace()
- Configured.resolve_merge_kwargs()
- Configured.update_config()
- ExecutionEngine.config
- ExecutionEngine.execute()
- ExecutionEngine.rec_state
- HasSettings.get_path_setting()
- HasSettings.get_path_settings()
- HasSettings.get_setting()
- HasSettings.get_settings()
- HasSettings.has_path_setting()
- HasSettings.has_path_settings()
- HasSettings.has_setting()
- HasSettings.has_settings()
- HasSettings.reset_settings()
- HasSettings.resolve_setting()
- HasSettings.resolve_settings_paths()
- HasSettings.set_settings()
- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.resolve_file_path()
- Pickleable.save()

### init_kwargs property

Keyword arguments used to initialize the pool.

### join_pool property

Whether to join the pool.

### pbar_kwargs property

Keyword arguments passed to get_pbar().

### pool_type property

Pool type.

### show_progress property

Whether to show the progress bar using get_pbar().

### sleep property

Number of seconds to sleep between checks.

### timeout property

Timeout.

## ProcessPoolEngine class

    ProcessPoolEngine(
        init_kwargs=None,
        timeout=None,
        **kwargs
    )

Class for executing functions using `ProcessPoolExecutor` from `concurrent.futures`.

For defaults, see `engines.processpool` in execution.

Superclasses

- Cacheable
- Chainable
- Comparable
- Configured
- ExecutionEngine
- HasSettings
- Pickleable
- Prettified

Inherited members

- Cacheable.get_ca_setup()
- Chainable.pipe()
- Configured.copy()
- Configured.equals()
- Configured.get_writeable_attrs()
- Configured.prettify()
- Configured.replace()
- Configured.resolve_merge_kwargs()
- Configured.update_config()
- ExecutionEngine.config
- ExecutionEngine.execute()
- ExecutionEngine.rec_state
- HasSettings.get_path_setting()
- HasSettings.get_path_settings()
- HasSettings.get_setting()
- HasSettings.get_settings()
- HasSettings.has_path_setting()
- HasSettings.has_path_settings()
- HasSettings.has_setting()
- HasSettings.has_settings()
- HasSettings.reset_settings()
- HasSettings.resolve_setting()
- HasSettings.resolve_settings_paths()
- HasSettings.set_settings()
- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.resolve_file_path()
- Pickleable.save()

### init_kwargs property

Keyword arguments used to initialize `ProcessPoolExecutor`.

### timeout property

Timeout.

## RayEngine class

    RayEngine(
        restart=None,
        reuse_refs=None,
        del_refs=None,
        shutdown=None,
        init_kwargs=None,
        remote_kwargs=None,
        **kwargs
    )

Class for executing functions in parallel using Ray.

For defaults, see `engines.ray` in execution.

Note

Ray spawns multiple processes as opposed to threads, so any argument and keyword argument must first be put into an object store to be shared. Make sure that the computation with `func` takes a considerable amount of time compared to this copying operation, otherwise there will be a little to no speedup.

Superclasses

- Cacheable
- Chainable
- Comparable
- Configured
- ExecutionEngine
- HasSettings
- Pickleable
- Prettified

Inherited members

- Cacheable.get_ca_setup()
- Chainable.pipe()
- Configured.copy()
- Configured.equals()
- Configured.get_writeable_attrs()
- Configured.prettify()
- Configured.replace()
- Configured.resolve_merge_kwargs()
- Configured.update_config()
- ExecutionEngine.config
- ExecutionEngine.execute()
- ExecutionEngine.rec_state
- HasSettings.get_path_setting()
- HasSettings.get_path_settings()
- HasSettings.get_setting()
- HasSettings.get_settings()
- HasSettings.has_path_setting()
- HasSettings.has_path_settings()
- HasSettings.has_setting()
- HasSettings.has_settings()
- HasSettings.reset_settings()
- HasSettings.resolve_setting()
- HasSettings.resolve_settings_paths()
- HasSettings.set_settings()
- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.resolve_file_path()
- Pickleable.save()

### del_refs property

Whether to explicitly delete the result object references.

### get_ray_refs class method

    RayEngine.get_ray_refs(
        funcs_args,
        reuse_refs=True,
        remote_kwargs=None
    )

Get result references by putting each argument and keyword argument into the object store and invoking the remote decorator on each function using Ray.

If `reuse_refs` is True, will generate one reference per unique object id.

### init_kwargs property

Keyword arguments passed to `ray.init`.

### remote_kwargs property

Keyword arguments passed to `ray.remote`.

### restart property

Whether to terminate the Ray runtime and initialize a new one.

### reuse_refs property

Whether to re-use function and object references, such that each unique object will be copied only once.

### shutdown property

Whether to True to terminate the Ray runtime upon the job end.

## SerialEngine class

    SerialEngine(
        progress_desc=None,
        show_progress=None,
        pbar_kwargs=None,
        clear_cache=None,
        collect_garbage=None,
        cooldown=None,
        **kwargs
    )

Class for executing functions sequentially.

For defaults, see `engines.serial` in execution.

Superclasses

- Cacheable
- Chainable
- Comparable
- Configured
- ExecutionEngine
- HasSettings
- Pickleable
- Prettified

Inherited members

- Cacheable.get_ca_setup()
- Chainable.pipe()
- Configured.copy()
- Configured.equals()
- Configured.get_writeable_attrs()
- Configured.prettify()
- Configured.replace()
- Configured.resolve_merge_kwargs()
- Configured.update_config()
- ExecutionEngine.config
- ExecutionEngine.execute()
- ExecutionEngine.rec_state
- HasSettings.get_path_setting()
- HasSettings.get_path_settings()
- HasSettings.get_setting()
- HasSettings.get_settings()
- HasSettings.has_path_setting()
- HasSettings.has_path_settings()
- HasSettings.has_setting()
- HasSettings.has_settings()
- HasSettings.reset_settings()
- HasSettings.resolve_setting()
- HasSettings.resolve_settings_paths()
- HasSettings.set_settings()
- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.resolve_file_path()
- Pickleable.save()

### clear_cache property

Whether to clear vectorbt's cache after each iteration.

If integer, do it once a number of calls.

### collect_garbage property

Whether to clear garbage after each iteration.

If integer, do it once a number of calls.

### cooldown property

Number of seconds to sleep after each call.

### pbar_kwargs property

Keyword arguments passed to get_pbar().

### progress_desc property

Sequence used to describe each iteration of the progress bar.

### show_progress property

Whether to show the progress bar using get_pbar().

## ThreadPoolEngine class

    ThreadPoolEngine(
        init_kwargs=None,
        timeout=None,
        **kwargs
    )

Class for executing functions using `ThreadPoolExecutor` from `concurrent.futures`.

For defaults, see `engines.threadpool` in execution.

Superclasses

- Cacheable
- Chainable
- Comparable
- Configured
- ExecutionEngine
- HasSettings
- Pickleable
- Prettified

Inherited members

- Cacheable.get_ca_setup()
- Chainable.pipe()
- Configured.copy()
- Configured.equals()
- Configured.get_writeable_attrs()
- Configured.prettify()
- Configured.replace()
- Configured.resolve_merge_kwargs()
- Configured.update_config()
- ExecutionEngine.config
- ExecutionEngine.execute()
- ExecutionEngine.rec_state
- HasSettings.get_path_setting()
- HasSettings.get_path_settings()
- HasSettings.get_setting()
- HasSettings.get_settings()
- HasSettings.has_path_setting()
- HasSettings.has_path_settings()
- HasSettings.has_setting()
- HasSettings.has_settings()
- HasSettings.reset_settings()
- HasSettings.resolve_setting()
- HasSettings.resolve_settings_paths()
- HasSettings.set_settings()
- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.resolve_file_path()
- Pickleable.save()

### init_kwargs property

Keyword arguments used to initialize `ThreadPoolExecutor`.

### timeout property

Timeout.

#### Cookie consent

We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better.

- Google Analytics
- GitHub

Manage settings