1. 02 Mar, 2023 35 commits
  2. 01 Mar, 2023 1 commit
  3. 28 Feb, 2023 4 commits
    • Roque's avatar
      58070276
    • Levin Zimmermann's avatar
      pandas: Fix unpickle np arrays with py2+pd>0.19.x · d223aede
      Levin Zimmermann authored
      Pandas 0.20.0 introduced a bug fix [1] which changed the behaviour of
      'DataFrame.to_records()', so that the resulting Record objects dtype names are
      unicodes if the data frames column names were unicode. Before this bug fix
      the dtype names were str, no matter whether the column names were str or unicode.
      
      Unfortunately np unpickle breaks if dtype names are unicode [2]. Since
      many of our data frame columns are unicode, loading arrays often
      fails. In python3 this isn't a problem anymore, so until then we fix
      this by introducing a simple monkey patch to pandas, which basically
      reverts the mentioned bug fix.
      
      [1] https://github.com/pandas-dev/pandas/issues/11879
      [2] Small example to reproduce this error:
      
      ''
      import os
      
      import numpy as np
      import pandas as pd
      
      r = pd.DataFrame({u'A':[1,2,3]}).to_records()
      a = np.ndarray(shape=r.shape, dtype=r.dtype.fields)
      p = "t"
      
      try:
        os.remove(p)
      except:
        pass
      
      with open(p, 'wb') as f:
        np.save(f, a)
      with open(p, 'rb') as f:
        np.load(f)
      ''
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      d223aede
    • Levin Zimmermann's avatar
      erp5_data_notebook += test un/pickle pd.DF=>np.rec · 611419d0
      Levin Zimmermann authored
      Add test which verifies that we can un/pickle 'pd.DataFrame' which were
      converted to 'np.recarray' via the '.to_records()' method.
      
      This test is necessary because it turns out that the combination of
      pandas >= 0.20.x + python2 doesn't support this functionality by
      default if the data frames column names are unicodes.
      
      Please see !1738 (comment 179298) for
      additional context.
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      611419d0
    • Levin Zimmermann's avatar
      Restricted: Allow newer pandas versions · f8301a19
      Levin Zimmermann authored
      There are newer pandas versions which still support python2. The latest
      one which still supports py2 (0.24.x) has initially been released in 2019
      and should therefore be quite stable and better supported. We should
      therefore prepare ERP5 so that it support those newer versions.
      
      This patch mostly fixes movements within the internal pandas structure.
      The patch still supports old object locations of previous pandas versions.
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      f8301a19