pyrseus.core.pickle.try_pickle_round_trip

pyrseus.core.pickle.try_pickle_round_trip(obj, *, dumps=<function _dumps>, protocol=-1, loads=<function _loads>, hide_main=True)[source]

Attempts to pickle and unpickle obj using dumps and loads, to help with testing and/or troubleshooting.

Example

Here’s an example of an object that can be pickled and unpickled without trouble:

>>> try_pickle_round_trip(42)
42

Here’s one that acts like it’s picklable and unpicklable:

>>> # simulate defining a trivial function in __main__, as happens
>>> # for scripts and notebooks.
>>> exec(
...     'def _func_in_main_for_try_pickle_round_trip(): return 42',
...     sys.modules['__main__'].__dict__,
... )
>>> # retrieve it
>>> func = sys.modules['__main__']._func_in_main_for_try_pickle_round_trip

>>> # technically, it's picklable and unpicklable, as long as the
>>> # unpickling happens in the same process:
>>> pickle.loads(pickle.dumps(func, -1))()
42

But if the unpickling happens in another process that doesn’t replicate the original’s __main__ module, then unpickling will fail, often with a confusing error message.

But with this function, we can simulate those unpickling problems, but at pickling time when they’re easier to troubleshoot:

>>> # Here it fails with a slightly better error message, and by
>>> # default using the pure Python pickler that's easier to debug
>>> # than the C one.
>>> try_pickle_round_trip(func)
Traceback (most recent call last):
...
  File ...pickle.py... in save_global
...
RuntimeError: Blocked attempt to read
_func_in_main_for_try_pickle_round_trip from __main__ while pickling
<function _func_in_main_for_try_pickle_round_trip at ...>.

>>> # Clean up
>>> del sys.modules['__main__']._func_in_main_for_try_pickle_round_trip

Warning

When hide_main=True, this function mutates global state by temporarily replacing sys.modules['__main__']. This function is threadsafe with respect to itself, but if some other concurrent code requires access to the __main__ module via sys.modules, then they may see unexpected exceptions. Such situations are rare.

Parameters:

  • obj – the object whose picklability is being tested

  • protocol – pickle protocol number to use. This is passed to dumps as a keyword argument.

  • dumps – a function that serializes objects to a pickle bytestring. This defaults to the pure Python pickle._dumps function. It is slower than the C pickle.dumps function, but it’s easier to troubleshoot with a Python debugger.

  • loads – a function that loads objects from a pickle bytestring. This defaults to the pure Python pickle._loads function. It is slower than the C pickle.loads function, but it’s easier to troubleshoot with a Python debugger.

  • hide_main – whether to hide the __main__ module during the pickle operation. This is done by temporarily replacing sys.modules['__main__'] with a proxy object that raises an exception on any attribute access. This makes it much easier to detect and troubleshoot pickling problems that arise from objects that belong to a script or notebook instead of a Python module, without needing to perform the unpickling in a separate process where it’s harder to debug.