• Jérome Perrin's avatar
    ERP5Type/XMLExportImport: use zodbpickle pickler for OrderedPickler · 00238dfe
    Jérome Perrin authored
    With upcoming ZODB 5, oids (used as persistent references in pickles)
    are no longer str as it use to be with ZODB 4, but instances of
    zodbpickle.binary, which with zodbpickle 1 are a subclass of str on
    python2.
    
    OrderedPickler was a subclass of pickle.Pickler, the pickler from standard
    library, but this pickler was not able to use a str subclass for persistent
    references, when pickles are loaded with noload method, persistent_load
    is called with `None` instead of the actual string subclass instance.
    This was problematic in the XMLExportImport handling of business templates,
    because ZODB.serialize.referencesf was unable to find persistent references.
    The error was:
    
        ZODB-5.6.0-py2.7.egg/ZODB/serialize.py", line 664, in referencesf
            assert isinstance(reference, list)
        AssertionError
    
    because the reference was None.
    
    zodbpickle 2 changed to make zodbpickle.binary implemented in C, which
    was failing earlier, because pickle.Pickle can not pickle these objects,
    failing in an error like this:
    
        lib/python2.7/copy_reg.py", line 70, in _reduce_ex
            raise TypeError, "can't pickle %s objects" % base.__name__
        TypeError: can't pickle binary objects
    
    This change also simplify our own implementation, by dropping jython
    support and calling save_dict on the super class instead of copying the
    implementation.
    
    Further references:
    
    - minimal script to reproduce the issues:
    
    ```python
    from __future__ import print_function
    import io
    import pickle
    
    import zodbpickle
    import zodbpickle.pickle
    import zodbpickle.fastpickle
    
    class ExternalObject(object):
      def __init__(self, oid):
        self.oid = oid
    
    def persistent_id(obj):
      if isinstance(obj, ExternalObject):
        return obj.oid
    
    def persistent_load(persid):
      print('persistent_load called with persid', repr(persid))
    
    o = ExternalObject(oid=zodbpickle.binary("binary persid"))
    
    for pickler_class in pickle.Pickler, zodbpickle.pickle.Pickler:
    
      f = io.BytesIO()
      p = pickler_class(f, 1)
      p.persistent_id = persistent_id
      p.dump(o)
    
      print('dump with pickler %s:\n  %r' % (pickler_class, f.getvalue()))
    
      # ZODB uses this unpickler
      up = zodbpickle.fastpickle.Unpickler(io.BytesIO(f.getvalue()))
      up.persistent_load = persistent_load
      up.noload()
    ```
    
    ```console
    $ python2 repro.py # with zodbpickle 1
    dump with pickler pickle.Pickler:
      'ccopy_reg\n_reconstructor\nq\x00(czodbpickle\nbinary\nq\x01c__builtin__\nstr\nq\x02U\rbinary persidq\x03tq\x04Rq\x05Q.'
    persistent_load called with persid None
    dump with pickler zodbpickle.pickle_2.Pickler:
      'U\rbinary persidq\x00Q.'
    persistent_load called with persid 'binary persid'
    ```
    
    ```console
    $ python2 repro.py # with zodbpickle 2
    Traceback (most recent call last):
      File "repro.py", line 45, in <module>
        p.dump(o)
      File ".../lib/python2.7/pickle.py", line 224, in dump
        self.save(obj)
      File ".../lib/python2.7/pickle.py", line 273, in save
        self.save_pers(pid)
      File ".../lib/python2.7/pickle.py", line 340, in save_pers
        self.save(pid)
      File ".../lib/python2.7/pickle.py", line 306, in save
        rv = reduce(self.proto)
      File ".../lib/python2.7/copy_reg.py", line 70, in _reduce_ex
        raise TypeError, "can't pickle %s objects" % base.__name__
    TypeError: can't pickle binary objects
    ```
    
    * ZODB change starting to use zodbpickle.binary instead of str:
    12ee41c4 (-ZODB now uses pickle protocol 3 for both Python 2 and Python 3., 2018-03-26)
    Since of 5.4.0 release
    
    * zodbpickle change starting to use C objects for zodbpickle.binary:
    bbef98c (Implement zodbpickle.binary in C for Py27., 2019-11-12)
    Since of 2.0.0 release
    00238dfe
XMLExportImport.py 8.36 KB