pyrseus.executors.pinline.PInlineExecutor

class pyrseus.executors.pinline.PInlineExecutor(**round_trip_kwargs)

Bases: InlineExecutor

__init__(**round_trip_kwargs)

Pickle-testing variant of InlineExecutor, using the built-in pickle module.

Summary

• Common Use Cases: troubleshooting pickling problems for other executors that use pickle for serialization.

• Concurrency: This is a non-concurrent, serial-only executor. All tasks are immediately run in the same process and thread they were submitted in.

• Exceptions: This executor has standard exception-handling semantics: all task-related exceptions are captured in the task’s future.

• Default max_workers: Not applicable.

• Pickling: This executor takes extra time to pickle and unpickle all tasks and their results. If you aren’t troubleshooting such issues and prefer lower overhead, consider using InlineExecutor instead.

Details

This variant pickles each submitted task and the task’s results, using the built-in pickle module. This is primarily useful for troubleshooting pickling problems occurring in multi-process executors, by performing the pickling and unpickling locally.

Here’s an example showing a pickling failure cause by trying to pickle a lambda with an executor type that uses pickle instead of cloudpickle.

Consider this lambda.

>>> import pickle
>>> needs_cloudpickle = lambda: "this only works with cpnocatch, not pnocatch"

It can’t be pickled with pickle.

>>> pickle.dumps(needs_cloudpickle, -1)
Traceback (most recent call last):
...
_pickle.PicklingError: Can't pickle ...

Since this class tests pickling with pickle, it correctly fails, as it would in a multi-process executor that uses pickle.

>>> with PInlineExecutor() as exe:
...     fut = exe.submit(needs_cloudpickle)
>>> # As with InlineExecutor, the exception is only propagated out
>>> # at result() time.
>>> print(fut.result())
Traceback (most recent call last):
...
_pickle.PicklingError: Can't pickle ...

Because the failure happens within the same process and thread as the submit call, it is easy to debug by tracing into it with a debugger.

See Pyrseus’ Executor Classes for a list of related executors.

param round_trip_kwargs:

keyword arguments passed to call_with_round_trip_pickling for each submitted task.

Methods

__init__(**round_trip_kwargs)	Pickle-testing variant of InlineExecutor, using the built-in pickle module.
map(fn, *iterables[, timeout, chunksize])	Returns an iterator equivalent to map(fn, iter).
shutdown(*args, **kwargs)	As a minor improvement on the base class' method, this override disallows submissions after a shutdown has been started.
submit(fcn, /, *args, **kwargs)	Immediately evaluates fcn(*args, **kwargs) and embeds the result in a Future.

__enter__()

__exit__(exc_type, exc_val, exc_tb)

map(fn, *iterables, timeout=None, chunksize=1)

Returns an iterator equivalent to map(fn, iter).

Parameters:

• fn – A callable that will take as many arguments as there are passed iterables.

• timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.

• chunksize – The size of the chunks the iterable will be broken into before being passed to a child process. This argument is only used by ProcessPoolExecutor; it is ignored by ThreadPoolExecutor.

Returns:

map(func, *iterables) but the calls may be evaluated out-of-order.

Return type:

An iterator equivalent to

Raises:

• TimeoutError – If the entire result iterator could not be generated before the given timeout.

• Exception – If fn(*args) raises for any values.

shutdown(*args, **kwargs)

As a minor improvement on the base class’ method, this override disallows submissions after a shutdown has been started. This can assist with finding bugs in user code.

submit(fcn, /, *args, **kwargs)

Immediately evaluates fcn(*args, **kwargs) and embeds the result in a Future. As with standard executors, any exceptions that occur are caught and recorded in that returned Future.