pyrseus.executors.cpinline.CpInlineExecutor

class pyrseus.executors.cpinline.CpInlineExecutor

Bases: InlineExecutor

Pickle-testing variant of InlineExecutor, using the 3rd party cloudpickle module.

Summary

• Common Use Cases: serially troubleshooting pickling problems for multi-process executors that use cloudpickle for serialization.

• Concurrency: This is a non-concurrent, serial-only executor. All tasks are immediately run in the same process and thread they were submitted in.

• Exceptions: This executor has standard exception-handling semantics: all task-related exceptions are captured in the task’s future.

• Default max_workers: Not applicable.

• Pickling: This executor takes extra time to pickle and unpickle all tasks and their results. If you aren’t troubleshooting such issues and prefer lower overhead, consider using InlineExecutor instead.

Details

This variant pickles each submitted task and the task’s results, using the cloudpickle module that understands things like lambdas. This is primarily useful for troubleshooting pickling problems occurring in multi-process executors, by performing the pickling and unpickling locally.

Consider this lambda.

>>> import pickle
>>> needs_cloudpickle = lambda: "this works if using cloudpickle for pickling"

It can’t be pickled with pickle.

>>> pickle.dumps(needs_cloudpickle, -1)
Traceback (most recent call last):
...
_pickle.PicklingError: Can't pickle ...

But it can be pickled with cloudpickle.

>>> print(pickle.loads(cloudpickle.dumps(needs_cloudpickle, -1))())
this works if using cloudpickle for pickling

Since this class uses cloudpickle for pickling, it works fine.

>>> with CpInlineExecutor() as exe:
...     fut = exe.submit(needs_cloudpickle)
...     print(fut.result())
this works if using cloudpickle for pickling

See Pyrseus’ Executor Classes for a list of related executors.

__init__()

An Executor that evaluates the task immediately upon submission, trapping exceptions like normal executors do.

Summary

• Common Use Cases:

• Light workloads: this executor is useful for avoiding concurrency overhead when running small batches of tasks. This lets developers avoid the alternative of rewriting all of their control flow to not use executors at all, just to get serial execution.

• Troubleshooting: since tasks are executed immediately and within the same thread, tracing through the task code in a debugger is trivially easy.

• Concurrency: This is a non-concurrent, serial-only executor. All tasks are immediately run in the same process and thread they were submitted in.

• Exceptions: This executor has standard exception-handling semantics: all task-related exceptions are captured in the task’s future.

• Default max_workers: Not applicable.

• Pickling: This executor does not perform any pickling.

Details

As with the only built-in within-process executor, ThreadPoolExecutor, arbitrary callables can be used, including non-picklable ones like lambdas:

>>> import os
>>> with InlineExecutor() as exe:
...     non_picklable = lambda: os.getpid()   # can't be pickled
...     fut = exe.submit(non_picklable)       # but works anyway
...     worker_pid_is_my_pid = fut.result()   # no pickling error
...     assert os.getpid() == worker_pid_is_my_pid

See Pyrseus’ Executor Classes for a list of related executors.

Methods

__init__()	An Executor that evaluates the task immediately upon submission, trapping exceptions like normal executors do.
map(fn, *iterables[, timeout, chunksize])	Returns an iterator equivalent to map(fn, iter).
shutdown(*args, **kwargs)	As a minor improvement on the base class' method, this override disallows submissions after a shutdown has been started.
submit(fcn, /, *args, **kwargs)	Immediately evaluates fcn(*args, **kwargs) and embeds the result in a Future.

__enter__()

__exit__(exc_type, exc_val, exc_tb)

map(fn, *iterables, timeout=None, chunksize=1)

Returns an iterator equivalent to map(fn, iter).

Parameters:

• fn – A callable that will take as many arguments as there are passed iterables.

• timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.

• chunksize – The size of the chunks the iterable will be broken into before being passed to a child process. This argument is only used by ProcessPoolExecutor; it is ignored by ThreadPoolExecutor.

Returns:

map(func, *iterables) but the calls may be evaluated out-of-order.

Return type:

An iterator equivalent to

Raises:

• TimeoutError – If the entire result iterator could not be generated before the given timeout.

• Exception – If fn(*args) raises for any values.

shutdown(*args, **kwargs)

As a minor improvement on the base class’ method, this override disallows submissions after a shutdown has been started. This can assist with finding bugs in user code.

submit(fcn, /, *args, **kwargs)

Immediately evaluates fcn(*args, **kwargs) and embeds the result in a Future. As with standard executors, any exceptions that occur are caught and recorded in that returned Future.