pyrseus.executors.cpprocess.CpProcessPoolExecutor

class pyrseus.executors.cpprocess.CpProcessPoolExecutor(max_workers=None, mp_context=None, initializer=None, initargs=(), *, max_tasks_per_child=None)[source]

Bases: ProcessPoolExecutor

A drop-in replacement for the built-in ProcessPoolExecutor that uses cloudpickle for pickling tasks and their return values, instead of pickle.

Summary

  • Common Use Cases: For the same use cases as concurrent.futures.ProcessPoolExecutor, but when users wish to use cloudpickle instead of pickle for serializing tasks and their results.

  • Concurrency: Each worker runs in its own process.

  • Exceptions: This executor has standard exception-handling semantics: all task-related exceptions are captured in the task’s future.

  • Default max_workers:* Uses get_num_available_cores instead of multiprocessing.cpu_count, respecting the CPU affinity mask when possible. Currently is unaware of cgroups constraints.

  • Pickling: cloudpickle

Details

Consider this lambda.

>>> needs_cloudpickle = lambda: 123

It can’t be pickled with pickle.

>>> import pickle
>>> pickle.dumps(needs_cloudpickle, -1)
Traceback (most recent call last):
...
_pickle.PicklingError: Can't pickle ...

And since the built-in ProcessPoolExecutor uses pickle, it can’t handle it:

>>> with ProcessPoolExecutor(1) as exe:
...     fut = exe.submit(needs_cloudpickle)
...     print(fut.result())
Traceback (most recent call last):
...
_pickle.PicklingError: Can't pickle ...

But CpProcessPoolExecutor is a thin wrapper around ProcessPoolExecutor that uses cloudpickle for serialization, so it works fine.

>>> with CpProcessPoolExecutor(1) as exe:
...     fut = exe.submit(needs_cloudpickle)
...     print(fut.result())
123

See Pyrseus’ Executor Classes for a list of related executors.

__init__(max_workers=None, mp_context=None, initializer=None, initargs=(), *, max_tasks_per_child=None)

Initializes a new ProcessPoolExecutor instance.

Parameters:

  • max_workers – The maximum number of processes that can be used to execute the given calls. If None or not given then as many worker processes will be created as the machine has processors.

  • mp_context – A multiprocessing context to launch the workers. This object should provide SimpleQueue, Queue and Process. Useful to allow specific multiprocessing start methods.

  • initializer – A callable used to initialize worker processes.

  • initargs – A tuple of arguments to pass to the initializer.

  • max_tasks_per_child – The maximum number of tasks a worker process can complete before it will exit and be replaced with a fresh worker process. The default of None means worker process will live as long as the executor. Requires a non-‘fork’ mp_context start method. When given, we default to using ‘spawn’ if no mp_context is supplied.

Methods

__init__([max_workers, mp_context, ...])

Initializes a new ProcessPoolExecutor instance.

map(fn, *iterables[, timeout, chunksize])

Returns an iterator equivalent to map(fn, iter).

shutdown([wait, cancel_futures])

Clean-up the resources associated with the Executor.

submit(fcn, /, *args, **kwargs)

Submits a callable to be executed with the given arguments.
__enter__()
__exit__(exc_type, exc_val, exc_tb)
map(fn, *iterables, timeout=None, chunksize=1)

Returns an iterator equivalent to map(fn, iter).

Parameters:

  • fn – A callable that will take as many arguments as there are passed iterables.

  • timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.

  • chunksize – If greater than one, the iterables will be chopped into chunks of size chunksize and submitted to the process pool. If set to one, the items in the list will be sent one at a time.

Returns:

map(func, *iterables) but the calls may be evaluated out-of-order.

Return type:

An iterator equivalent to

Raises:

  • TimeoutError – If the entire result iterator could not be generated before the given timeout.

  • Exception – If fn(*args) raises for any values.

shutdown(wait=True, *, cancel_futures=False)

Clean-up the resources associated with the Executor.

It is safe to call this method several times. Otherwise, no other methods can be called after this one.

Parameters:

  • wait – If True then shutdown will not return until all running futures have finished executing and the resources used by the executor have been reclaimed.

  • cancel_futures – If True then shutdown will cancel all pending futures. Futures that are completed or running will not be cancelled.

submit(fcn, /, *args, **kwargs)[source]

Submits a callable to be executed with the given arguments.

Schedules the callable to be executed as fn(*args, **kwargs) and returns a Future instance representing the execution of the callable.

Returns:

A Future representing the given call.