Skip to content

execution module

Engines for executing functions.

build_serial_chunk function

build_serial_chunk(
    funcs_args
)

Build a serial chunk.

execute function

execute(
    funcs_args,
    engine='serial',
    n_calls=None,
    min_size=None,
    n_chunks=None,
    chunk_len=None,
    chunk_meta=None,
    distribute=None,
    warmup=None,
    in_chunk_order=False,
    pre_execute_func=None,
    pre_execute_kwargs=None,
    pre_chunk_func=None,
    pre_chunk_kwargs=None,
    post_chunk_func=None,
    post_chunk_kwargs=None,
    post_execute_func=None,
    post_execute_kwargs=None,
    post_execute_on_sorted=False,
    show_progress=None,
    progress_desc=None,
    pbar_kwargs=None,
    template_context=None,
    engine_kwargs=None,
    **kwargs
)

Execute using an engine.

Supported values for engine:

Can execute per chunk if chunk_meta is provided. Otherwise, if any of n_chunks and chunk_len are set, passes them to yield_chunk_meta() to generate chunk_meta. Arguments n_chunks and chunk_len can be set globally in the engine-specific settings. Set n_chunks and chunk_len to 'auto' to set them to the number of cores.

If distribute is "calls", distributes calls within each chunk. If indices in chunk_meta are perfectly sorted and funcs_args is an iterable, iterates over funcs_args to avoid converting it into a list. Otherwise, iterates over chunk_meta. If in_chunk_order is True, returns the outputs in the order they appear in chunk_meta. Otherwise, always returns them in the same order as in funcs_args.

If distribute is "chunks", distributes chunks. For this, executes calls within each chunk serially using execute_serially(). Also, compresses each chunk such that each unique function, positional argument, and keyword argument is serialized only once.

If funcs_args is a custom template, substitutes it once chunk_meta is established. Use template_context as an additional context. All the resolved functions and arguments will be immediately passed to the executor.

If pre_chunk_func is not None, calls the function before processing a chunk. If it returns anything other than None, the returned object will be appended to the outputs and the chunk won't be executed. This enables use cases such as caching. If post_chunk_func is not None, calls the function after processing the chunk. It should return either None to keep the old call outputs, or return new ones. Will also substitute any templates in pre_chunk_kwargs and post_chunk_kwargs and pass them as keyword arguments. The following additional arguments are available in the contexts: the index of the current chunk chunk_idx, the list of call indices call_indices in the chunk, the list of call outputs call_outputs returned by executing the chunk (only for post_chunk_func), and whether the chunk was executed chunk_executed or otherwise returned by pre_chunk_func (only for post_chunk_func).

Note

The both callbacks above are effective only when distribute is "calls" and chunking is enabled.

If pre_execute_func is not None, calls the function before processing all calls. Should return nothing (None). Will also substitute any templates in post_execute_kwargs and pass them as keyword arguments. The following additional arguments are available in the context: the number of chunks n_chunks.

If post_execute_func is not None, calls the function after processing all calls. Will also substitute any templates in post_execute_kwargs and pass them as keyword arguments. Should return either None to keep the default outputs or return the new ones. The following additional arguments are available in the context: the number of chunks n_chunks and the generated flattened list of outputs outputs. If post_execute_on_sorted is True, will run the callback after sorting the call indices.

Info

Chunks are processed sequentially, while functions within each chunk can be processed distributively.

Supported engines can be found in engines in execution.

execute_serially function

execute_serially(
    funcs_args,
    id_objs
)

Execute serially.

pass_kwargs_as_args function

pass_kwargs_as_args(
    func,
    args,
    kwargs
)

Helper function for pathos.pools.ParallelPool.

DaskEngine class

DaskEngine(
    compute_kwargs=None,
    **kwargs
)

Class for executing functions in parallel using Dask.

For defaults, see engines.dask in execution.

Note

Use multi-threading mainly on numeric code that releases the GIL (like NumPy, Pandas, Scikit-Learn, Numba).

Superclasses

Inherited members

compute_kwargs property

Keyword arguments passed to dask.compute.

ExecutionEngine class

ExecutionEngine(
    **config
)

Abstract class for executing functions.

Superclasses

Inherited members

Subclasses

execute method

ExecutionEngine.execute(
    funcs_args,
    n_calls=None
)

Run an iterable of tuples out of a function, arguments, and keyword arguments.

Provide n_calls in case funcs_args is a generator and the underlying engine needs it.

MpireEngine class

MpireEngine(
    init_kwargs=None,
    apply_kwargs=None,
    **kwargs
)

Class for executing functions using WorkerPool from mpire.

For defaults, see engines.mpire in execution.

Superclasses

Inherited members

apply_kwargs property

Keyword arguments passed to WorkerPool.async_apply.

init_kwargs property

Keyword arguments used to initialize WorkerPool.

PathosEngine class

PathosEngine(
    pool_type=None,
    init_kwargs=None,
    timeout=None,
    sleep=None,
    show_progress=None,
    pbar_kwargs=None,
    join_pool=None,
    **kwargs
)

Class for executing functions using pathos.

For defaults, see engines.pathos in execution.

Superclasses

Inherited members

init_kwargs property

Keyword arguments used to initialize the pool.

join_pool property

Whether to join the pool.

pbar_kwargs property

Keyword arguments passed to get_pbar().

pool_type property

Pool type.

show_progress property

Whether to show the progress bar using get_pbar().

sleep property

Number of seconds to sleep between checks.

timeout property

Timeout.

ProcessPoolEngine class

ProcessPoolEngine(
    init_kwargs=None,
    timeout=None,
    **kwargs
)

Class for executing functions using ProcessPoolExecutor from concurrent.futures.

For defaults, see engines.processpool in execution.

Superclasses

Inherited members

init_kwargs property

Keyword arguments used to initialize ProcessPoolExecutor.

timeout property

Timeout.

RayEngine class

RayEngine(
    restart=None,
    reuse_refs=None,
    del_refs=None,
    shutdown=None,
    init_kwargs=None,
    remote_kwargs=None,
    **kwargs
)

Class for executing functions in parallel using Ray.

For defaults, see engines.ray in execution.

Note

Ray spawns multiple processes as opposed to threads, so any argument and keyword argument must first be put into an object store to be shared. Make sure that the computation with func takes a considerable amount of time compared to this copying operation, otherwise there will be a little to no speedup.

Superclasses

Inherited members

del_refs property

Whether to explicitly delete the result object references.

get_ray_refs class method

RayEngine.get_ray_refs(
    funcs_args,
    reuse_refs=True,
    remote_kwargs=None
)

Get result references by putting each argument and keyword argument into the object store and invoking the remote decorator on each function using Ray.

If reuse_refs is True, will generate one reference per unique object id.

init_kwargs property

Keyword arguments passed to ray.init.

remote_kwargs property

Keyword arguments passed to ray.remote.

restart property

Whether to terminate the Ray runtime and initialize a new one.

reuse_refs property

Whether to re-use function and object references, such that each unique object will be copied only once.

shutdown property

Whether to True to terminate the Ray runtime upon the job end.

SerialEngine class

SerialEngine(
    progress_desc=None,
    show_progress=None,
    pbar_kwargs=None,
    clear_cache=None,
    collect_garbage=None,
    cooldown=None,
    **kwargs
)

Class for executing functions sequentially.

For defaults, see engines.serial in execution.

Superclasses

Inherited members

clear_cache property

Whether to clear vectorbt's cache after each iteration.

If integer, do it once a number of calls.

collect_garbage property

Whether to clear garbage after each iteration.

If integer, do it once a number of calls.

cooldown property

Number of seconds to sleep after each call.

pbar_kwargs property

Keyword arguments passed to get_pbar().

progress_desc property

Sequence used to describe each iteration of the progress bar.

show_progress property

Whether to show the progress bar using get_pbar().

ThreadPoolEngine class

ThreadPoolEngine(
    init_kwargs=None,
    timeout=None,
    **kwargs
)

Class for executing functions using ThreadPoolExecutor from concurrent.futures.

For defaults, see engines.threadpool in execution.

Superclasses

Inherited members

init_kwargs property

Keyword arguments used to initialize ThreadPoolExecutor.

timeout property

Timeout.