1. 02 Mar, 2023 1 commit
  2. 01 Mar, 2023 5 commits
    • Levin Zimmermann's avatar
      pandas: Fix unpickle np arrays with py2+pd>0.19.x · 1730ba8e
      Levin Zimmermann authored
      Pandas 0.20.0 introduced a bug fix [1] which changed the behaviour of
      'DataFrame.to_records()', so that the resulting Record objects dtype names are
      unicodes if the data frames column names were unicode. Before this bug fix
      the dtype names were str, no matter whether the column names were str or unicode.
      
      Unfortunately np unpickle breaks if dtype names are unicode [2]. Since
      many of our data frame columns are unicode, loading arrays often
      fails. In python3 this isn't a problem anymore, so until then we fix
      this by introducing a simple monkey patch to pandas, which basically
      reverts the mentioned bug fix.
      
      [1] https://github.com/pandas-dev/pandas/issues/11879
      [2] Small example to reproduce this error:
      
      ''
      import os
      
      import numpy as np
      import pandas as pd
      
      r = pd.DataFrame({u'A':[1,2,3]}).to_records()
      a = np.ndarray(shape=r.shape, dtype=r.dtype.fields)
      p = "t"
      
      try:
        os.remove(p)
      except:
        pass
      
      with open(p, 'wb') as f:
        np.save(f, a)
      with open(p, 'rb') as f:
        np.load(f)
      ''
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      1730ba8e
    • Levin Zimmermann's avatar
      erp5_data_notebook += test un/pickle pd.DF=>np.rec · 6180f1af
      Levin Zimmermann authored
      Add test which verifies that we can un/pickle 'pd.DataFrame' which were
      converted to 'np.recarray' via the '.to_records()' method.
      
      This test is necessary because it turns out that the combination of
      pandas >= 0.20.x + python2 doesn't support this functionality by
      default if the data frames column names are unicodes.
      
      Please see !1738 (comment 179298) for
      additional context.
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      6180f1af
    • Levin Zimmermann's avatar
      Restricted: Allow newer pandas versions · 6d3cf100
      Levin Zimmermann authored
      There are newer pandas versions which still support python2. The latest
      one which still supports py2 (0.24.x) has initially been released in 2019
      and should therefore be quite stable and better supported. We should
      therefore prepare ERP5 so that it support those newer versions.
      
      This patch mostly fixes movements within the internal pandas structure.
      The patch still supports old object locations of previous pandas versions.
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      6d3cf100
    • Levin Zimmermann's avatar
      restricted: Allow usage of resampler · 573ef107
      Levin Zimmermann authored
      We already use '.resample.mean()' in some projects.
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      573ef107
    • Levin Zimmermann's avatar
      erp5_core_test: restricted: Test pandas resampler · c5e0d11c
      Levin Zimmermann authored
      In some projects we use pandas resampler inside 'portal_callables'
      (so with restricted Python). But we miss tests which check
      whether pandas resamplers are allowed inside the restricted Python.
      Due to this lack of tests we won't notice whether the usage of pandas
      resamplers still work with a newer pandas version. With this patch
      we can recognize if an internal change of pandas makes the resampler unusuable
      in restricted Python.
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      c5e0d11c
  3. 15 Feb, 2023 2 commits
  4. 30 Jan, 2023 21 commits
  5. 27 Jan, 2023 3 commits
  6. 26 Jan, 2023 1 commit
  7. 25 Jan, 2023 4 commits
    • Xiaowu Zhang's avatar
      erp5_web_renderjs_ui_test: fix broken tests · ee8ca461
      Xiaowu Zhang authored
      See merge request nexedi/erp5!1731
      ee8ca461
    • Xiaowu Zhang's avatar
    • Xiaowu Zhang's avatar
      43300d92
    • Jérome Perrin's avatar
      Compatibility with DateTime >= 3 · fd737b5f
      Jérome Perrin authored
      On the way to Zope 4, the biggest change was DateTime >= 3, which has these main differences:
      
       - `DateTime` instances are new style classes
       - `DateTime.__eq__` now take into the timezone into account
       - `DateTime` supports timezone naive ( https://docs.python.org/3/library/datetime.html#aware-and-naive-objects ) - actually this is since DateTime 2.12 , but since we fully patched the method where the parsing happens, ERP5's patched DateTime never used timezone naive dates.
       - pickles are also different.
      
      These are changes to prepare ERP5 code base to support DateTime >= 3, while keeping compatibility with the previous behavior, this means that we keep patching to keep the same behavior as DateTime 2, so that `DateTime.__eq__` ignores timezone difference and `DateTime` does not support timezone naive dates - these two different behavior seemed to introduce too much breakages, especially in project code, with no immediate benefit for our usages. For now, while ERP5 is still based on Zope 2 / DateTime 2 the fact that DateTime are new style classes has no impact, this will only impact once we switch to Zope 4.
      
      This also change the code to adjust the part where we were depending on implementation details of DateTime and where the details changed on DateTime 3, sometimes exposing higher level API, like the new `timeZoneContext` to use in tests.
      
      
      
      See merge request nexedi/erp5!1593
      fd737b5f
  8. 24 Jan, 2023 3 commits