Forums

Pickle problems with Jupyter Notebook

I got 99 problems and now a pickle is one!

Getting: MaybeEncodingError: Error sending result: '[<backtrader.cerebro.OptReturn object at 0x7f02b6787080>]'. Reason: 'PicklingError("Can't pickle <function <lambda> at 0x7f02ec32f620>: attribute lookup <lambda> on jupyter_client.session failed",)'

This is using the package backtrader, and I suspect it has something to do with multithreading, although I'm certainly not advanced enough to know!

This problem does not happen when I'm running this same code on my laptop.

Full traceback:

MaybeEncodingError                        Traceback (most recent call last)
<ipython-input-6-1a90ce74d998> in <module>()
----> 1 opt_runs = cerebro.run()

/home/tjw0000/.local/lib/python3.6/site-packages/backtrader/cerebro.py in run(self, **kwargs)
   1141 
   1142             pool = multiprocessing.Pool(self.p.maxcpus or None)
-> 1143             for r in pool.imap(self, iterstrats):
   1144                 self.runstrats.append(r)
   1145                 for cb in self.optcbs:

/usr/lib/python3.6/multiprocessing/pool.py in next(self, timeout)
    697         if success:
    698             return value
--> 699         raise value
    700 
    701     __next__ = next                    # XXX

Thanks for any help!

deleted-user-2746501 | 16 posts | July 13, 2018, 5:10 p.m. | permalink

Hmm. We did have a couple of problems on one of our notebook servers about the time when you posted this -- could you try again now and see if the issue has gone away?

Staff giles | 12096 posts | PythonAnywhere staff | July 13, 2018, 5:42 p.m. | permalink

Same error unfortunately. I believe it has to do with multi-threading... 

---------------------------------------------------------------------------
MaybeEncodingError                        Traceback (most recent call last)
<ipython-input-7-de2ece101a81> in <module>()
      1 get_ipython().magic('debug')
----> 2 opt_runs = cerebro.run()

/home/tjw0000/.local/lib/python3.6/site-packages/backtrader/cerebro.py in run(self, **kwargs)
   1141 
   1142             pool = multiprocessing.Pool(self.p.maxcpus or None)
-> 1143             for r in pool.imap(self, iterstrats):
   1144                 self.runstrats.append(r)
   1145                 for cb in self.optcbs:

/usr/lib/python3.6/multiprocessing/pool.py in next(self, timeout)
    697         if success:
    698             return value
--> 699         raise value
    700 
    701     __next__ = next                    # XXX

MaybeEncodingError: Error sending result: '[<backtrader.cerebro.OptReturn object at 0x7f79dbcae2b0>]'. Reason: 'PicklingError("Can't pickle <function <lambda> at 0x7f7a115f1620>: attribute lookup <lambda> on jupyter_client.session failed",)'
deleted-user-2746501 | 16 posts | July 14, 2018, 12:02 a.m. | permalink

It looks to me like you're trying to pickle something that's not pickleable. In particular, it looks like there's some sort of lambda that you're trying to include in your pickle and pickle cannot handle lambdas.

Staff glenn | 9718 posts | PythonAnywhere staff | July 14, 2018, 10:01 a.m. | permalink

This exact code runs on my laptop, so I think it might have something to do with jupyter_client.session .. is it possible I have a different version of jupyter?

deleted-user-2746501 | 16 posts | July 14, 2018, 6:49 p.m. | permalink

If you use a virtualenv, you can install whichever version of jupyter that you like.

Staff glenn | 9718 posts | PythonAnywhere staff | July 15, 2018, 5:27 p.m. | permalink