1. 04 Dec, 2023 2 commits
    • Levin Zimmermann's avatar
      Always set soft fds limit to hard fds limit · 08be8e99
      Levin Zimmermann authored
      This patch is specifically for the WWM/wind project. It unconditionally
      sets the soft limit of open file descriptors to the hard limit. It is
      'unconditionally' in two meanings:
      
      1. Always set it and don't depend on instance configuration (via
         'instance.json). We certainly want this in the WWM/wind instance and
         adding the option to SlapOS would lead to a lot of copy-pasted SlapOS
         instance files in the WWM project, which are noisy, difficult to
         maintain and tedious to create.
      
      2. Always set it and don't depend on the used server (Medusa or WSGI).
         In the upstream patches nexedi/erp5@39369169
         and nexedi/erp5@c42c1d38
         the option is only made available when using WSGI. But in our production
         instance we may want to switch back to ZServer/Medusa, which is why
         we need an ERP5 which supports the option independent from the used
         server.
      08be8e99
    • Levin Zimmermann's avatar
      fixup! ERP5/zopewsgi: Add option to set soft limit of open file descriptors to hard limit · a6f6c48f
      Levin Zimmermann authored
      In nexedi/erp5@39369169 the new option
      'nofile' was added to set the soft limit of open file descriptors to its hard limit.
      In nexedi/slapos!1451 (comment 193296) it was criticized that
      the name of the option - 'nofile' - is too vague and in nexedi/slapos!1465 a new
      name for the option was searched. Finally we agreed on the more explicit 'with-max-rlimit-nofile'.
      a6f6c48f
  2. 27 Oct, 2023 1 commit
    • Levin Zimmermann's avatar
      ERP5/zopewsgi: Add option to set soft limit of open file descriptors to hard limit · b2e8e825
      Levin Zimmermann authored
      The default soft limit of open file descriptors is usually set to 1024
      in order to avoid breaking old software which still uses select. In many
      projects we may need a higher limit: particularly in Wendelin based
      projects we easily reach this limit. Before this patch it was therefore
      necessary to either patch ERP5 in the project specific SR or to manually
      increase the limit of the zope processes (or the parent supervisor) with
      a tool like prlimit [1]. With this patch it becomes possible to increase the
      soft limit to the hard limit with a command line argument of the zopewsgi bin.
      This simplifies setting the soft limit for any Wendelin project.
      
      [1] https://man7.org/linux/man-pages/man1/prlimit.1.html
      
      /reviewed-by @vpelletier, @jerome
      /reviewed-on nexedi/erp5!1827
      
      ---
      
      NOTE wind commit:
      
      We cherry-pick commit nexedi/erp5@39369169
      for Mic.Wind, to automatically avoid the open file descriptors limit
      issue.
      b2e8e825
  3. 03 Mar, 2023 2 commits
  4. 02 Mar, 2023 2 commits
  5. 01 Mar, 2023 1 commit
  6. 28 Feb, 2023 6 commits
    • Roque's avatar
      58070276
    • Levin Zimmermann's avatar
      pandas: Fix unpickle np arrays with py2+pd>0.19.x · d223aede
      Levin Zimmermann authored
      Pandas 0.20.0 introduced a bug fix [1] which changed the behaviour of
      'DataFrame.to_records()', so that the resulting Record objects dtype names are
      unicodes if the data frames column names were unicode. Before this bug fix
      the dtype names were str, no matter whether the column names were str or unicode.
      
      Unfortunately np unpickle breaks if dtype names are unicode [2]. Since
      many of our data frame columns are unicode, loading arrays often
      fails. In python3 this isn't a problem anymore, so until then we fix
      this by introducing a simple monkey patch to pandas, which basically
      reverts the mentioned bug fix.
      
      [1] https://github.com/pandas-dev/pandas/issues/11879
      [2] Small example to reproduce this error:
      
      ''
      import os
      
      import numpy as np
      import pandas as pd
      
      r = pd.DataFrame({u'A':[1,2,3]}).to_records()
      a = np.ndarray(shape=r.shape, dtype=r.dtype.fields)
      p = "t"
      
      try:
        os.remove(p)
      except:
        pass
      
      with open(p, 'wb') as f:
        np.save(f, a)
      with open(p, 'rb') as f:
        np.load(f)
      ''
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      d223aede
    • Levin Zimmermann's avatar
      erp5_data_notebook += test un/pickle pd.DF=>np.rec · 611419d0
      Levin Zimmermann authored
      Add test which verifies that we can un/pickle 'pd.DataFrame' which were
      converted to 'np.recarray' via the '.to_records()' method.
      
      This test is necessary because it turns out that the combination of
      pandas >= 0.20.x + python2 doesn't support this functionality by
      default if the data frames column names are unicodes.
      
      Please see !1738 (comment 179298) for
      additional context.
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      611419d0
    • Levin Zimmermann's avatar
      Restricted: Allow newer pandas versions · f8301a19
      Levin Zimmermann authored
      There are newer pandas versions which still support python2. The latest
      one which still supports py2 (0.24.x) has initially been released in 2019
      and should therefore be quite stable and better supported. We should
      therefore prepare ERP5 so that it support those newer versions.
      
      This patch mostly fixes movements within the internal pandas structure.
      The patch still supports old object locations of previous pandas versions.
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      f8301a19
    • Levin Zimmermann's avatar
      restricted: Allow usage of resampler · b5f3e9d8
      Levin Zimmermann authored
      We already use '.resample.mean()' in some projects.
      
      /reviewed-on !1738
      /reviewed-by @jerome @klaus
      b5f3e9d8
    • Levin Zimmermann's avatar
      erp5_core_test: restricted: Test pandas resampler · a5696ae9
      Levin Zimmermann authored
      In some projects we use pandas resampler inside 'portal_callables'
      (so with restricted Python). But we miss tests which check
      whether pandas resamplers are allowed inside the restricted Python.
      Due to this lack of tests we won't notice whether the usage of pandas
      resamplers still work with a newer pandas version. With this patch
      we can recognize if an internal change of pandas makes the resampler unusuable
      in restricted Python.
      
      /reviewed-on nexedi/erp5!1738
      /reviewed-by @jerome @klaus
      a5696ae9
  7. 27 Feb, 2023 4 commits
  8. 24 Feb, 2023 4 commits
  9. 23 Feb, 2023 1 commit
  10. 22 Feb, 2023 6 commits
  11. 21 Feb, 2023 5 commits
  12. 18 Feb, 2023 1 commit
  13. 17 Feb, 2023 5 commits