Skip to content
New: Star-import, signal unraveling, Python 3.12 support, and more

#
chunking module

Utilities for chunking.

## chunked function

    chunked(
        *args,
        chunker_cls=None,
        size=None,
        min_size=None,
        n_chunks=None,
        chunk_len=None,
        chunk_meta=None,
        prepend_chunk_meta=None,
        skip_single_chunk=None,
        arg_take_spec=None,
        template_context=None,
        merge_func=None,
        merge_kwargs=None,
        return_raw_chunks=False,
        silence_warnings=None,
        disable=None,
        forward_kwargs_as=None,
        execute_kwargs=None,
        **kwargs
    )

Decorator that chunks the inputs of a function using Chunker.

Returns a new function with the same signature as the passed one.

Each option can be modified in the `options` attribute of the wrapper function or directly passed as a keyword argument with a leading underscore.

Keyword arguments `**kwargs` are passed directly to execute().

Chunking can be disabled using `disable` argument. Additionally, the entire wrapping mechanism can be disabled by using the global setting `disable_wrapping` (=> returns the wrapped function).

Usage

For testing purposes, let's divide the input array into 2 chunks and compute the mean in a sequential manner:

    >>> from vectorbtpro import *

    >>> @vbt.chunked(
    ...     n_chunks=2,
    ...     size=vbt.LenSizer(arg_query='a'),
    ...     arg_take_spec=dict(a=vbt.ChunkSlicer())
    ... )
    ... def f(a):
    ...     return np.mean(a)

    >>> f(np.arange(10))
    [2.0, 7.0]

Same can be done using annotations:

    >>> @vbt.chunked(n_chunks=2)
    ... def f(a: vbt.LenSizer() | vbt.ChunkSlicer()):
    ...     return np.mean(a)

    >>> f(np.arange(10))
    [2.0, 7.0]

Sizer can be omitted most of the time:

    >>> @vbt.chunked(n_chunks=2)
    ... def f(a: vbt.ChunkSlicer()):
    ...     return np.mean(a)

    >>> f(np.arange(10))
    [2.0, 7.0]

Another way is by using specialized Chunker subclasses that depend on the type of the argument:

    >>> @vbt.chunked(n_chunks=2)
    ... def f(a: vbt.ChunkedArray()):
    ...     return np.mean(a)

    >>> f(np.arange(10))

Also, instead of specifying the chunk taking specification beforehand, it can be passed dynamically by wrapping each value to be chunked with Chunked or any of its subclasses:

    >>> @vbt.chunked(n_chunks=2)
    ... def f(a):
    ...     return np.mean(a)

    >>> f(vbt.ChunkedArray(np.arange(10)))
    [2.0, 7.0]

The chunked() function is a decorator that takes `f` and creates a function that splits passed arguments, runs each chunk using an engine, and optionally, merges the results. It has the same signature as the original function:

    >>> f
    <function __main__.f(a)>

We can change any option at any time:

    >>> # Change the option directly on the function
    >>> f.options.n_chunks = 3

    >>> f(np.arange(10))
    [1.5, 5.0, 8.0]

    >>> # Pass a new option with a leading underscore
    >>> f(np.arange(10), _n_chunks=4)
    [1.0, 4.0, 6.5, 8.5]

When we run the wrapped function, it first generates a list of chunk metadata of type ChunkMeta. Chunk metadata contains the chunk index that can be used to split any input:

    >>> list(vbt.yield_chunk_meta(n_chunks=2))
    [ChunkMeta(uuid='84d64eed-fbac-41e7-ad61-c917e809b3b8', idx=0, start=None, end=None, indices=None),
     ChunkMeta(uuid='577817c4-fdee-4ceb-ab38-dcd663d9ab11', idx=1, start=None, end=None, indices=None)]

Additionally, it may contain the start and end index of the space we want to split. The space can be defined by the length of an input array, for example. In our case:

    >>> list(vbt.yield_chunk_meta(n_chunks=2, size=10))
    [ChunkMeta(uuid='c1593842-dc31-474c-a089-e47200baa2be', idx=0, start=0, end=5, indices=None),
     ChunkMeta(uuid='6d0265e7-1204-497f-bc2c-c7b7800ec57d', idx=1, start=5, end=10, indices=None)]

If we know the size of the space in advance, we can pass it as an integer constant. Otherwise, we need to tell chunked() to derive the size from the inputs dynamically by passing any subclass of Sizer. In the example above, we instruct the wrapped function to derive the size from the length of the input array `a`.

Once all chunks are generated, the wrapped function attempts to split inputs into chunks. The specification for this operation can be provided by the `arg_take_spec` argument, which in most cases is a dictionary of ChunkTaker instances keyed by the input name. Here's an example of a complex specification:

    >>> arg_take_spec = dict(
    ...     a=vbt.ChunkSelector(),
    ...     args=vbt.ArgsTaker(
    ...         None,
    ...         vbt.ChunkSelector()
    ...     ),
    ...     b=vbt.SequenceTaker([
    ...         None,
    ...         vbt.ChunkSelector()
    ...     ]),
    ...     kwargs=vbt.KwargsTaker(
    ...         c=vbt.MappingTaker(dict(
    ...             d=vbt.ChunkSelector(),
    ...             e=None
    ...         ))
    ...     )
    ... )

    >>> @vbt.chunked(
    ...     n_chunks=vbt.LenSizer(arg_query='a'),
    ...     arg_take_spec=arg_take_spec
    ... )
    ... def f(a, *args, b=None, **kwargs):
    ...     return a + sum(args) + sum(b) + sum(kwargs['c'].values())

    >>> f([1, 2, 3], 10, [1, 2, 3], b=(100, [1, 2, 3]), c=dict(d=[1, 2, 3], e=1000))
    [1114, 1118, 1122]

After splitting all inputs into chunks, the wrapped function forwards them to the engine function. The engine argument can be either the name of a supported engine, or a callable. Once the engine has finished all tasks and returned a list of results, we can merge them back using `merge_func`:

    >>> @vbt.chunked(
    ...     n_chunks=2,
    ...     size=vbt.LenSizer(arg_query='a'),
    ...     arg_take_spec=dict(a=vbt.ChunkSlicer()),
    ...     merge_func="concat"
    ... )
    ... def f(a):
    ...     return a

    >>> f(np.arange(10))
    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

The same using annotations:

    >>> @vbt.chunked(n_chunks=2)
    ... def f(a: vbt.ChunkSlicer()) -> vbt.MergeFunc("concat"):
    ...     return a

    >>> f(np.arange(10))
    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Instead of (or in addition to) specifying `arg_take_spec`, we can define our function with the first argument being `chunk_meta` to be able to split the arguments during the execution. The chunked() decorator will automatically recognize and replace it with the actual ChunkMeta object:

    >>> @vbt.chunked(
    ...     n_chunks=2,
    ...     size=vbt.LenSizer(arg_query='a'),
    ...     arg_take_spec=dict(a=None),
    ...     merge_func="concat"
    ... )
    ... def f(chunk_meta, a):
    ...     return a[chunk_meta.start:chunk_meta.end]

    >>> f(np.arange(10))
    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

This may be a good idea in multi-threading, but a bad idea in multi-processing.

The same can be accomplished by using templates (here we tell chunked() to not replace the first argument by setting `prepend_chunk_meta` to False):

    >>> @vbt.chunked(
    ...     n_chunks=2,
    ...     size=vbt.LenSizer(arg_query='a'),
    ...     arg_take_spec=dict(a=None),
    ...     merge_func="concat",
    ...     prepend_chunk_meta=False
    ... )
    ... def f(chunk_meta, a):
    ...     return a[chunk_meta.start:chunk_meta.end]

    >>> f(vbt.Rep('chunk_meta'), np.arange(10))
    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

Templates in arguments are substituted right before taking a chunk from them.

Keyword arguments to the engine can be provided using `execute_kwargs`:

    >>> @vbt.chunked(
    ...     n_chunks=2,
    ...     size=vbt.LenSizer(arg_query='a'),
    ...     arg_take_spec=dict(a=vbt.ChunkSlicer()),
    ...     show_progress=True
    ... )
    ... def f(a):
    ...     return np.mean(a)

    >>> f(np.arange(10))
    100% |█████████████████████████████████| 2/2 [00:00<00:00, 81.11it/s]
    [2.0, 7.0]

## resolve_chunked function

    resolve_chunked(
        func,
        option=None,
        **kwargs
    )

Decorate with chunked() based on an option.

## resolve_chunked_option function

    resolve_chunked_option(
        option=None
    )

Return keyword arguments for chunked().

`option` can be:

- True: Chunk using default settings
- None or False: Do not chunk
- string: Use `option` as the name of an execution engine (see execute())
- dict: Use `option` as keyword arguments passed to chunked()

For defaults, see `option` in chunking.

## specialize_chunked_option function

    specialize_chunked_option(
        option=None,
        **kwargs
    )

Resolve `option` and merge it with `kwargs` if it's not None so the dict can be passed as an option to other functions.

## yield_chunk_meta function

    yield_chunk_meta(
        size=None,
        min_size=None,
        n_chunks=None,
        chunk_len=None
    )

Yield meta of each successive chunk from a sequence with a number of elements.

If both `n_chunks` and `chunk_len` are None (after resolving them from settings), sets `n_chunks` to the number of cores.

## ArgChunkMeta class

    ArgChunkMeta(
        arg_query=None
    )

Class for generating chunk metadata from an argument.

Method generated by attrs for class ArgGetter.

Superclasses

- ArgGetter
- ChunkMetaGenerator

Inherited members

- ArgGetter.arg_query
- ArgGetter.get_arg()
- ChunkMetaGenerator.get_chunk_meta()

Subclasses

- LenChunkMeta

## ArgGetter class

    ArgGetter(
        arg_query=None
    )

Class for getting an argument from annotated arguments.

Method generated by attrs for class ArgGetter.

Subclasses

- ArgChunkMeta
- ArgSizer
- GroupIdxsMapper
- GroupLensMapper

### arg_query class variable

Query for annotated argument to derive the size from.

### get_arg method

    ArgGetter.get_arg(
        ann_args
    )

Get argument using match_ann_arg().

## ArgSizer class

    ArgSizer(
        arg_query=None,
        single_type=None
    )

Class for getting the size from an argument.

Method generated by attrs for class ArgSizer.

Superclasses

- Annotatable
- ArgGetter
- Sizer

Inherited members

- ArgGetter.arg_query
- ArgGetter.get_arg()
- Sizer.apply()
- Sizer.get_size()

Subclasses

- CountSizer
- GroupLensSizer
- LenSizer
- ShapeSizer

### single_type class variable

One or multiple types to consider as a single value.

## ArgsTaker class

    ArgsTaker(
        *args,
        single_type=None,
        ignore_none=True,
        mapper=None
    )

Class for taking from a variable arguments container.

Method generated by attrs for class ChunkTaker.

Superclasses

- Annotatable
- ChunkTaker
- ContainerTaker
- SequenceTaker

Inherited members

- ChunkTaker.apply()
- ChunkTaker.should_take()
- ContainerTaker.check_cont_take_spec()
- ContainerTaker.get_size()
- SequenceTaker.adapt_cont_take_spec()
- SequenceTaker.cont_take_spec
- SequenceTaker.ignore_none
- SequenceTaker.mapper
- SequenceTaker.single_type
- SequenceTaker.suggest_size()
- SequenceTaker.take()

## ArraySelector class

    ArraySelector(
        single_type=None,
        ignore_none=True,
        mapper=None,
        keep_dims=False,
        axis=None
    )

Class for selecting one element from an array's axis based on the chunk index.

Method generated by attrs for class ShapeSelector.

Superclasses

- Annotatable
- AxisSpecifier
- ChunkSelector
- ChunkTaker
- DimRetainer
- ShapeSelector

Inherited members

- ChunkSelector.suggest_size()
- ChunkTaker.apply()
- ChunkTaker.should_take()
- ShapeSelector.axis
- ShapeSelector.get_size()
- ShapeSelector.ignore_none
- ShapeSelector.keep_dims
- ShapeSelector.mapper
- ShapeSelector.single_type
- ShapeSelector.take()

Subclasses

- FlexArraySelector

## ArraySizer class

    ArraySizer(
        arg_query=None,
        single_type=None,
        axis=None
    )

Class for getting the size from the length of an axis in an array.

Method generated by attrs for class ShapeSizer.

Superclasses

- Annotatable
- ArgGetter
- ArgSizer
- AxisSpecifier
- ShapeSizer
- Sizer

Inherited members

- ArgGetter.get_arg()
- ArgSizer.apply()
- ShapeSizer.arg_query
- ShapeSizer.axis
- ShapeSizer.get_obj_size()
- ShapeSizer.get_size()
- ShapeSizer.single_type

Subclasses

- FlexArraySizer

## ArraySlicer class

    ArraySlicer(
        single_type=None,
        ignore_none=True,
        mapper=None,
        axis=None
    )

Class for slicing multiple elements from an array's axis based on the chunk range.

Method generated by attrs for class ShapeSlicer.

Superclasses

- Annotatable
- AxisSpecifier
- ChunkSlicer
- ChunkTaker
- ShapeSlicer

Inherited members

- ChunkTaker.apply()
- ChunkTaker.should_take()
- ChunkTaker.suggest_size()
- ShapeSlicer.axis
- ShapeSlicer.get_size()
- ShapeSlicer.ignore_none
- ShapeSlicer.mapper
- ShapeSlicer.single_type
- ShapeSlicer.take()

Subclasses

- FlexArraySlicer

## AxisSpecifier class

    AxisSpecifier(
        axis=None
    )

Class with an attribute for specifying an axis.

Method generated by attrs for class AxisSpecifier.

Subclasses

- ShapeSelector
- ShapeSizer
- ShapeSlicer

### axis class variable

Axis of the argument to take from.

## ChunkMapper class

    ChunkMapper(
        should_cache=True,
        chunk_meta_cache=NOTHING
    )

Abstract class for mapping chunk metadata.

Implements the abstract ChunkMapper.map() method.

Supports caching of each pair of incoming and outgoing ChunkMeta instances.

Note

Use ChunkMapper.apply() instead of ChunkMapper.map().

Method generated by attrs for class ChunkMapper.

Subclasses

- GroupIdxsMapper
- GroupLensMapper

### apply method

    ChunkMapper.apply(
        chunk_meta,
        **kwargs
    )

Apply the mapper.

### chunk_meta_cache class variable

Cache for outgoing ChunkMeta instances keyed by UUID of the incoming ones.

### map method

    ChunkMapper.map(
        chunk_meta,
        **kwargs
    )

Abstract method for mapping chunk metadata.

Takes the chunk metadata of type ChunkMeta and returns a new chunk metadata of the same type.

### should_cache class variable

Whether should cache.

## ChunkMeta class

    ChunkMeta(
        uuid,
        idx,
        start,
        end,
        indices
    )

Class that represents a chunk metadata.

Method generated by attrs for class ChunkMeta.

### end class variable

End of the chunk range (excluding). Can be None.

### idx class variable

Chunk index.

### indices class variable

Indices included in the chunk range. Can be None.

Has priority over ChunkMeta.start and ChunkMeta.end.

### start class variable

Start of the chunk range (including). Can be None.

### uuid class variable

Unique identifier of the chunk.

Used for caching.

## ChunkMetaGenerator class

    ChunkMetaGenerator()

Abstract class for generating chunk metadata from annotated arguments.

Subclasses

- ArgChunkMeta

### get_chunk_meta method

    ChunkMetaGenerator.get_chunk_meta(
        ann_args,
        **kwargs
    )

Get chunk metadata.

## ChunkSelector class

    ChunkSelector(
        single_type=None,
        ignore_none=True,
        mapper=None,
        keep_dims=False
    )

Class for selecting one element based on the chunk index.

Method generated by attrs for class ChunkSelector.

Superclasses

- Annotatable
- ChunkTaker
- DimRetainer

Inherited members

- ChunkTaker.apply()
- ChunkTaker.get_size()
- ChunkTaker.ignore_none
- ChunkTaker.mapper
- ChunkTaker.should_take()
- ChunkTaker.single_type
- ChunkTaker.suggest_size()
- ChunkTaker.take()
- DimRetainer.keep_dims

Subclasses

- ShapeSelector

## ChunkSlicer class

    ChunkSlicer(
        single_type=None,
        ignore_none=True,
        mapper=None
    )

Class for slicing multiple elements based on the chunk range.

Method generated by attrs for class ChunkTaker.

Superclasses

- Annotatable
- ChunkTaker

Inherited members

- ChunkTaker.apply()
- ChunkTaker.get_size()
- ChunkTaker.ignore_none
- ChunkTaker.mapper
- ChunkTaker.should_take()
- ChunkTaker.single_type
- ChunkTaker.suggest_size()
- ChunkTaker.take()

Subclasses

- CountAdapter
- GroupLensSlicer
- GroupMapSlicer
- ShapeSlicer

## ChunkTaker class

    ChunkTaker(
        single_type=None,
        ignore_none=True,
        mapper=None
    )

Abstract class for taking one or more elements based on the chunk index or range.

Note

Use ChunkTaker.apply() instead of ChunkTaker.take().

Method generated by attrs for class ChunkTaker.

Superclasses

- Annotatable

Subclasses

- ChunkSelector
- ChunkSlicer
- ContainerTaker

### apply method

    ChunkTaker.apply(
        obj,
        chunk_meta,
        **kwargs
    )

Apply the taker.

### get_size method

    ChunkTaker.get_size(
        obj,
        **kwargs
    )

Get the actual size of the argument.

### ignore_none class variable

Whether to ignore None.

### mapper class variable

Chunk mapper of type ChunkMapper.

### should_take method

    ChunkTaker.should_take(
        obj,
        chunk_meta,
        **kwargs
    )

Check whether to take a chunk or leave the argument as it is.

### single_type class variable

One or multiple types to consider as a single value.

### suggest_size method

    ChunkTaker.suggest_size(
        obj,
        **kwargs
    )

Suggest a global size based on the argument's size.

### take method

    ChunkTaker.take(
        obj,
        chunk_meta,
        **kwargs
    )

Abstract method for taking subset of data.

Takes the argument object, the chunk meta (tuple out of the index, start index, and end index of the chunk), and other keyword arguments passed down the stack, such as `chunker` and `silence_warnings`.

## Chunkable class

    Chunkable()

Abstract class representing a value and a chunk taking specification.

Subclasses

- Chunked

### get_take_spec method

    Chunkable.get_take_spec()

Get the chunk taking specification.

### get_value method

    Chunkable.get_value()

Get the value.

## Chunked class

    Chunked(
        *args,
        **kwargs
    )

Class representing a chunkable value.

Can take a variable number of keyword arguments, which will be used as Chunked.take_spec_kwargs.

Superclasses

- Annotatable
- Chunkable

Inherited members

- Chunkable.get_take_spec()
- Chunkable.get_value()

Subclasses

- ChunkedArray
- ChunkedCount
- ChunkedFlexArray
- ChunkedGroupLens
- ChunkedGroupMap
- ChunkedShape

### check_value method

    Chunked.check_value()

Check whether value is missing.

### resolve_take_spec method

    Chunked.resolve_take_spec()

Resolve `take_spec`.

### select class variable

Whether to chunk by selection.

### take_spec class variable

Chunk taking specification.

### take_spec_kwargs class variable

Keyword arguments passed to the respective ChunkTaker subclass.

If Chunked.take_spec is an instance rather than a class, will "evolve" it.

### take_spec_missing property

Check whether Chunked.take_spec is missing.

### value class variable

Value.

### value_missing property

Check whether Chunked.value is missing.

## ChunkedArray class

    ChunkedArray(
        *args,
        **kwargs
    )

Class representing a chunkable array.

Superclasses

- Annotatable
- Chunkable
- Chunked

Inherited members

- Chunked.check_value()
- Chunked.get_take_spec()
- Chunked.get_value()
- Chunked.resolve_take_spec()
- Chunked.select
- Chunked.take_spec
- Chunked.take_spec_kwargs
- Chunked.take_spec_missing
- Chunked.value
- Chunked.value_missing

### flex class variable

Whether the array is flexible.

## ChunkedCount class

    ChunkedCount(
        *args,
        **kwargs
    )

Class representing a chunkable count.

Superclasses

- Annotatable
- Chunkable
- Chunked

Inherited members

- Chunked.check_value()
- Chunked.get_take_spec()
- Chunked.get_value()
- Chunked.resolve_take_spec()
- Chunked.select
- Chunked.take_spec
- Chunked.take_spec_kwargs
- Chunked.take_spec_missing
- Chunked.value
- Chunked.value_missing

## ChunkedShape class

    ChunkedShape(
        *args,
        **kwargs
    )

Class representing a chunkable shape.

Superclasses

- Annotatable
- Chunkable
- Chunked

Inherited members

- Chunked.check_value()
- Chunked.get_take_spec()
- Chunked.get_value()
- Chunked.resolve_take_spec()
- Chunked.select
- Chunked.take_spec
- Chunked.take_spec_kwargs
- Chunked.take_spec_missing
- Chunked.value
- Chunked.value_missing

## Chunker class

    Chunker(
        func,
        size=None,
        min_size=None,
        n_chunks=None,
        chunk_len=None,
        chunk_meta=None,
        prepend_chunk_meta=None,
        skip_single_chunk=None,
        arg_take_spec=None,
        template_context=None,
        merge_func=None,
        merge_kwargs=None,
        return_raw_chunks=None,
        silence_warnings=None,
        disable=None,
        forward_kwargs_as=None,
        execute_kwargs=None,
        **kwargs
    )

Class responsible for chunking arguments of a function and running the function.

Superclasses

- Cacheable
- Chainable
- Comparable
- Configured
- HasSettings
- Pickleable
- Prettified

Inherited members

- Cacheable.get_ca_setup()
- Chainable.pipe()
- Configured.config
- Configured.copy()
- Configured.equals()
- Configured.get_writeable_attrs()
- Configured.prettify()
- Configured.rec_state
- Configured.replace()
- Configured.resolve_merge_kwargs()
- Configured.update_config()
- HasSettings.get_path_setting()
- HasSettings.get_path_settings()
- HasSettings.get_setting()
- HasSettings.get_settings()
- HasSettings.has_path_setting()
- HasSettings.has_path_settings()
- HasSettings.has_setting()
- HasSettings.has_settings()
- HasSettings.reset_settings()
- HasSettings.resolve_setting()
- HasSettings.resolve_settings_paths()
- HasSettings.set_settings()
- Pickleable.decode_config()
- Pickleable.decode_config_node()
- Pickleable.dumps()
- Pickleable.encode_config()
- Pickleable.encode_config_node()
- Pickleable.file_exists()
- Pickleable.getsize()
- Pickleable.load()
- Pickleable.loads()
- Pickleable.modify_state()
- Pickleable.resolve_file_path()
- Pickleable.save()

### adapt_ann_args class method

    Chunker.adapt_ann_args(
        ann_args
    )

Adapt annotated arguments.

### arg_take_spec property

See `yield_arg_chunks`.

### chunk_len property

See Chunker.get_chunk_meta_from_args().

### chunk_meta property

See Chunker.get_chunk_meta_from_args().

### disable property

Whether to disable chunking entirely.

### execute_kwargs property

Keyword arguments passed to execute().

### fill_arg_take_spec class method

    Chunker.fill_arg_take_spec(
        arg_take_spec,
        ann_args
    )

Fill the chunk taking specification with None to avoid warnings.

### find_take_spec class method

    Chunker.find_take_spec(
        i,
        ann_arg_name,
        ann_arg,
        arg_take_spec
    )

Resolve the specification for an argument.

### forward_kwargs_as property

Map to rename keyword arguments.

Can also pass any variable from the scope of Chunker.run()

### func property

Function.

### get_chunk_meta_from_args class method

    Chunker.get_chunk_meta_from_args(
        ann_args,
        size=None,
        min_size=None,
        n_chunks=None,
        chunk_len=None,
        chunk_meta=None,
        **kwargs
    )

Get chunk metadata from annotated arguments.

Args

`ann_args` :`dict`
  Arguments annotated with annotate_args().
`size` :`int`, Sizer, `or callable`

  Size of the space to split.

  Can be an integer, an instance of Sizer, or a callable taking the annotated arguments and returning an integer.

`min_size` :`int`
  If `size` is lower than this number, returns a single chunk.
`n_chunks` :`int`, Sizer, `or callable`

  Number of chunks.

  Can be an integer, an instance of Sizer, or a callable taking the annotated arguments and other keyword arguments and returning an integer.

`chunk_len` :`int`, Sizer, `or callable`

  Length of each chunk.

  Can be an integer, an instance of Sizer, or a callable taking the annotated arguments and returning an integer.

`chunk_meta` :`iterable` of ChunkMeta, ChunkMetaGenerator, `or callable`

  Chunk meta.

  Can be an iterable of ChunkMeta, an instance of ChunkMetaGenerator, or a callable taking the annotated arguments and other arguments and returning an iterable.

`**kwargs`
  Other keyword arguments passed to any callable.

### merge_func property

Merging function.

Resolved using resolve_merge_func().

### merge_kwargs property

Keyword arguments passed to the merging function.

### min_size property

See Chunker.get_chunk_meta_from_args().

### n_chunks property

See Chunker.get_chunk_meta_from_args().

### parse_sizer_from_func class method

    Chunker.parse_sizer_from_func(
        func
    )

Parse the sizer from a function.

### parse_spec_from_annotations class method

    Chunker.parse_spec_from_annotations(
        annotations
    )

Parse the chunk taking specification from annotations.

### parse_spec_from_args class method

    Chunker.parse_spec_from_args(
        ann_args
    )

Parse the chunk taking specification from (annotated) arguments.

### parse_spec_from_func class method

    Chunker.parse_spec_from_func(
        func
    )

Parse the chunk taking specification from a function.

### prepend_chunk_meta property

Whether to prepend an instance of ChunkMeta to the arguments.

If None, prepends automatically if the first argument is named 'chunk_meta'.

### resolve_take_spec class method

    Chunker.resolve_take_spec(
        take_spec
    )

Resolve the chunk taking specification.

### return_raw_chunks property

Whether to return chunks in a raw format.

### run method

    Chunker.run(
        *args,
        **kwargs
    )

Chunk arguments and run the function.

Does the following:

1. Generates chunk metadata by passing `n_chunks`, `size`, `min_size`, `chunk_len`, and `chunk_meta` to Chunker.get_chunk_meta_from_args().
2. Splits arguments and keyword arguments by passing chunk metadata, `arg_take_spec`, and `template_context` to Chunker.yield_arg_chunks(), which yields one chunk at a time.
3. Executes all chunks by passing `**execute_kwargs` to execute().
4. Optionally, post-processes and merges the results by passing them and `**merge_kwargs` to `merge_func`.

For defaults, see chunking.

### silence_warnings property

Whether to silence any warnings.

### size property

See yield_chunk_meta().

### skip_single_chunk property

Whether to execute the function directly if there's only one chunk.

### suggest_size class method

    Chunker.suggest_size(
        ann_args,
        arg_take_spec,
        **kwargs
    )

Suggest a global size given the annotated arguments and the chunk taking specification.

### take_from_arg class method

    Chunker.take_from_arg(
        arg,
        take_spec,
        chunk_meta,
        **kwargs
    )

Take from the argument given the specification `take_spec`.

If `take_spec` is None or it's an instance of NotChunked, returns the original object. Otherwise, must be an instance of ChunkTaker.

`**kwargs` are passed to ChunkTaker.apply().

### take_from_args class method

    Chunker.take_from_args(
        ann_args,
        arg_take_spec,
        chunk_meta,
        silence_warnings=False,
        **kwargs
    )

Take from each in the annotated arguments given the specification using Chunker.take_from_arg().

Additionally, passes to Chunker.take_from_arg() as keyword arguments `ann_args` and `arg_take_spec`.

`arg_take_spec` must be a dictionary, with keys being argument positions or names as generated by annotate_args(). For values, see Chunker.take_from_arg().

Returns arguments and keyword arguments that can be directly passed to the function using `func(*args, **kwargs)`.

### template_context property

Template context.

Any template in both `execute_kwargs` and `merge_kwargs` will be substituted. You can use the keys `ann_args`, `chunk_meta`, `arg_take_spec`, and `funcs_args` to be replaced by the actual objects.

### yield_arg_chunks class method

    Chunker.yield_arg_chunks(
        func,
        ann_args,
        chunk_meta,
        arg_take_spec=None,
        template_context=None,
        **kwargs
    )

Split annotated arguments into chunks using Chunker.take_from_args() and yield each chunk.

Args

`func` :`callable`
  Callable.
`ann_args` :`dict`
  Arguments annotated with annotate_args().
`chunk_meta` :`iterable` of ChunkMeta
  Chunk metadata.
`arg_take_spec` :`mapping`, `sequence`, `callable`, `or CustomTemplate`

  Chunk taking specification.

  Can be a dictionary (see Chunker.take_from_args()), or a sequence that will be converted into a dictionary. If a callable, will be called instead of Chunker.take_from_args(), thus it must have the same arguments apart from `arg_take_spec`.

`template_context` :`mapping`
  Mapping to replace templates in arguments and specification.
`**kwargs`
  Keyword arguments passed to Chunker.take_from_args() or to `arg_take_spec` if it's a callable.

## ContainerTaker class

    ContainerTaker(
        cont_take_spec=None,
        single_type=None,
        ignore_none=True,
        mapper=None
    )

Class for taking from a container with other chunk takers.

Accepts the specification of the container.

Method generated by attrs for class ChunkTaker.

Superclasses

- Annotatable
- ChunkTaker

Inherited members

- ChunkTaker.apply()
- ChunkTaker.get_size()
- ChunkTaker.ignore_none
- ChunkTaker.mapper
- ChunkTaker.should_take()
- ChunkTaker.single_type
- ChunkTaker.suggest_size()
- ChunkTaker.take()

Subclasses

- MappingTaker
- SequenceTaker

### check_cont_take_spec method

    ContainerTaker.check_cont_take_spec()

Check that ContainerTaker.cont_take_spec is not None.

### cont_take_spec class variable

Specification of the container.

## CountAdapter class

    CountAdapter(
        single_type=None,
        ignore_none=True,
        mapper=None
    )

Class for adapting a count based on the chunk range.

Method generated by attrs for class ChunkTaker.

Superclasses

- Annotatable
- ChunkSlicer
- ChunkTaker

Inherited members

- ChunkSlicer.get_size()
- ChunkSlicer.ignore_none
- ChunkSlicer.mapper
- ChunkSlicer.single_type
- ChunkSlicer.take()
- ChunkTaker.apply()
- ChunkTaker.should_take()
- ChunkTaker.suggest_size()

## CountSizer class

    CountSizer(
        arg_query=None,
        single_type=None
    )

Class for getting the size from a count.

Method generated by attrs for class ArgSizer.

Superclasses

- Annotatable
- ArgGetter
- ArgSizer
- Sizer

Inherited members

- ArgGetter.get_arg()
- ArgSizer.apply()
- ArgSizer.arg_query
- ArgSizer.get_size()
- ArgSizer.single_type

### get_obj_size class method

    CountSizer.get_obj_size(
        obj,
        single_type=None
    )

Get size of an object.

## DimRetainer class

    DimRetainer(
        keep_dims=False
    )

Class with an attribute for retaining dimensions.

Method generated by attrs for class DimRetainer.

Subclasses

- ChunkSelector

### keep_dims class variable

Whether to retain dimensions.

## KwargsTaker class

    KwargsTaker(
        single_type=None,
        ignore_none=True,
        mapper=None,
        **kwargs
    )

Class for taking from a variable keyword arguments container.

Method generated by attrs for class ChunkTaker.

Superclasses

- Annotatable
- ChunkTaker
- ContainerTaker
- MappingTaker

Inherited members

- ChunkTaker.apply()
- ChunkTaker.should_take()
- ContainerTaker.check_cont_take_spec()
- ContainerTaker.get_size()
- MappingTaker.adapt_cont_take_spec()
- MappingTaker.cont_take_spec
- MappingTaker.ignore_none
- MappingTaker.mapper
- MappingTaker.single_type
- MappingTaker.suggest_size()
- MappingTaker.take()

## LenChunkMeta class

    LenChunkMeta(
        arg_query=None
    )

Class for generating chunk metadata from a sequence of chunk lengths.

Method generated by attrs for class ArgGetter.

Superclasses

- ArgChunkMeta
- ArgGetter
- ChunkMetaGenerator

Inherited members

- ArgChunkMeta.arg_query
- ArgChunkMeta.get_chunk_meta()
- ArgGetter.get_arg()

## LenSizer class

    LenSizer(
        arg_query=None,
        single_type=None
    )

Class for getting the size from the length of an argument.

Method generated by attrs for class ArgSizer.

Superclasses

- Annotatable
- ArgGetter
- ArgSizer
- Sizer

Inherited members

- ArgGetter.get_arg()
- ArgSizer.apply()
- ArgSizer.arg_query
- ArgSizer.get_size()
- ArgSizer.single_type

### get_obj_size class method

    LenSizer.get_obj_size(
        obj,
        single_type=None
    )

Get size of an object.

## MappingTaker class

    MappingTaker(
        cont_take_spec=None,
        single_type=None,
        ignore_none=True,
        mapper=None
    )

Class for taking from a mapping container.

Calls Chunker.take_from_arg() on each element.

Method generated by attrs for class ChunkTaker.

Superclasses

- Annotatable
- ChunkTaker
- ContainerTaker

Inherited members

- ChunkTaker.apply()
- ChunkTaker.should_take()
- ChunkTaker.suggest_size()
- ContainerTaker.check_cont_take_spec()
- ContainerTaker.cont_take_spec
- ContainerTaker.get_size()
- ContainerTaker.ignore_none
- ContainerTaker.mapper
- ContainerTaker.single_type
- ContainerTaker.take()

Subclasses

- KwargsTaker

### adapt_cont_take_spec method

    MappingTaker.adapt_cont_take_spec(
        obj
    )

Prepare the specification of the container to the object.

## NotChunked class

    NotChunked()

Class that represents an argument that shouldn't be chunked.

Method generated by attrs for class NotChunked.

Superclasses

- Annotatable

## SequenceTaker class

    SequenceTaker(
        cont_take_spec=None,
        single_type=None,
        ignore_none=True,
        mapper=None
    )

Class for taking from a sequence container.

Calls Chunker.take_from_arg() on each element.

Method generated by attrs for class ChunkTaker.

Superclasses

- Annotatable
- ChunkTaker
- ContainerTaker

Inherited members

- ChunkTaker.apply()
- ChunkTaker.should_take()
- ChunkTaker.suggest_size()
- ContainerTaker.check_cont_take_spec()
- ContainerTaker.cont_take_spec
- ContainerTaker.get_size()
- ContainerTaker.ignore_none
- ContainerTaker.mapper
- ContainerTaker.single_type
- ContainerTaker.take()

Subclasses

- ArgsTaker

### adapt_cont_take_spec method

    SequenceTaker.adapt_cont_take_spec(
        obj
    )

Prepare the specification of the container to the object.

## ShapeSelector class

    ShapeSelector(
        single_type=None,
        ignore_none=True,
        mapper=None,
        keep_dims=False,
        axis=None
    )

Class for selecting one element from a shape's axis based on the chunk index.

Method generated by attrs for class ShapeSelector.

Superclasses

- Annotatable
- AxisSpecifier
- ChunkSelector
- ChunkTaker
- DimRetainer

Inherited members

- AxisSpecifier.axis
- ChunkSelector.get_size()
- ChunkSelector.ignore_none
- ChunkSelector.keep_dims
- ChunkSelector.mapper
- ChunkSelector.single_type
- ChunkSelector.suggest_size()
- ChunkSelector.take()
- ChunkTaker.apply()
- ChunkTaker.should_take()

Subclasses

- ArraySelector

## ShapeSizer class

    ShapeSizer(
        arg_query=None,
        single_type=None,
        axis=None
    )

Class for getting the size from the length of an axis in a shape.

Method generated by attrs for class ShapeSizer.

Superclasses

- Annotatable
- ArgGetter
- ArgSizer
- AxisSpecifier
- Sizer

Inherited members

- ArgGetter.get_arg()
- ArgSizer.apply()
- ArgSizer.arg_query
- ArgSizer.get_size()
- ArgSizer.single_type
- AxisSpecifier.axis

Subclasses

- ArraySizer

### get_obj_size class method

    ShapeSizer.get_obj_size(
        obj,
        axis,
        single_type=None
    )

Get size of an object.

## ShapeSlicer class

    ShapeSlicer(
        single_type=None,
        ignore_none=True,
        mapper=None,
        axis=None
    )

Class for slicing multiple elements from a shape's axis based on the chunk range.

Method generated by attrs for class ShapeSlicer.

Superclasses

- Annotatable
- AxisSpecifier
- ChunkSlicer
- ChunkTaker

Inherited members

- AxisSpecifier.axis
- ChunkSlicer.get_size()
- ChunkSlicer.ignore_none
- ChunkSlicer.mapper
- ChunkSlicer.single_type
- ChunkSlicer.take()
- ChunkTaker.apply()
- ChunkTaker.should_take()
- ChunkTaker.suggest_size()

Subclasses

- ArraySlicer

## Sizer class

    Sizer()

Abstract class for getting the size from annotated arguments.

Note

Use Sizer.apply() instead of Sizer.get_size().

Superclasses

- Annotatable

Subclasses

- ArgSizer

### apply method

    Sizer.apply(
        ann_args,
        **kwargs
    )

Apply the sizer.

### get_size method

    Sizer.get_size(
        ann_args,
        **kwargs
    )

Get the size given the annotated arguments.

#### Cookie consent

We use cookies to recognize your repeated visits and preferences, as well as to measure the effectiveness of our documentation and whether users find what they're searching for. With your consent, you're helping us to make our documentation better.

- Google Analytics
- GitHub

Manage settings