1. 13 Jun, 2022 5 commits
    • Jérome Perrin's avatar
      erp5_ui_test: move translation utility script to erp5_ui_test_core · f3c1434a
      Jérome Perrin authored
      so that they can be reused by other UI tests business templates without
      having to install the full erp5_ui_test
      f3c1434a
    • Jérome Perrin's avatar
      trade: Consider movement dates to select trade condition for price · 7193b12d
      Jérome Perrin authored
      using movement date rather than order date is more correct
      7193b12d
    • Jérome Perrin's avatar
      composition: only consider validated when looking up effective models · 5d929056
      Jérome Perrin authored
      composition API, which is used to select the "most applicable" model
      (Trade Condition, Transformation, Pay Sheet Model, etc) with the same
      reference was only discarding deleted and validated models, but it
      was selecting draft models.
      
      This leads to this kind of problem:
       - A validated model exist, with an effective date
       - A new model is made by cloning the validated one, edited to set
        effective date from now and validated
       - The first validated model is also cloned by mistake, but not
        validated
      In such case, the model cloned by mistake might be used, which from
      user point of view is wrong, because user validated a model, so if
      the system use a draft instead, that's a very wrong behaviour.
      
      This change require models to be validated for the composition API to
      work, many tests were updated to validate the model (either in the test
      or in the data business template used by the test).
      
      This change might also produce different results in production, because
      it was possible that some draft models were selected before, now they
      will no longer be selected. To check for potential side effects, we
      recommend looking for all draft models with a reference, examine them
      one by one and validate the model if it's a case where user forgot to
      validate.
      5d929056
    • Vincent Pelletier's avatar
      Products.CMFActivity: Unconditionally change user in dummyGroupMethod · 0e57df28
      Vincent Pelletier authored
      Since
        commit f363ac65
        Author: Vincent Pelletier <vincent@nexedi.com>
        Date:   Wed Mar 23 15:55:46 2022 +0900
      
            Products.CMFActivity.ActivityTool: Store user object in activity.
      user_name on newly created activities is always None. As a result,
      activities using dummyGroupMethod are executed within the security
      context which spawns the group, which is System Processes, instead of
      the user which spawned each activity.
      Add a unittest for this.
      0e57df28
    • Vincent Pelletier's avatar
      Products.CMFActivity: Fix user name in activity tracking log messages · 984f7f13
      Vincent Pelletier authored
      This was broken by:
        commit f363ac65
        Author: Vincent Pelletier <vincent@nexedi.com>
        Date:   Wed Mar 23 15:55:46 2022 +0900
      
            Products.CMFActivity.ActivityTool: Store user object in activity.
      as user_name becomes always None.
      984f7f13
  2. 09 Jun, 2022 6 commits
    • Jérome Perrin's avatar
      Enable coding style on erp5_payroll · 34d50a78
      Jérome Perrin authored
      See merge request nexedi/erp5!1637
      34d50a78
    • Jérome Perrin's avatar
      core: repair ERP5Site_reindexLatestIndexedObjects · e83e7960
      Jérome Perrin authored
      This script stopped working since Catalog changed to be ERP5 document,
      because it was calling the unindex method of the catalog (which
      itself is indexable like any other ERP5 document). Update to use the
      uncatalogObject which is the method to unindex a document by uid.
      
      Even though it was somehow working before, it was not really correct
      in selecting objects, because it was using - operator on TIMESTAMP
      column, which is not computing a difference in seconds as this script
      was expecting. See for example https://stackoverflow.com/a/24504132/7607763
      or the example below for an explanation. Instead, use TIMESTAMPADD
      to compute the start timestamp only once and use >= operator, which
      works as expected.
      
      This query was also sorting by indexation_timestamp, which does not
      use an index. Remove the sort because it's not really needed.
      
      Excluding reserved path was also not needed, we no longer use these
      since 69aefdff (ZSQLCatalog: Drop support for "reserved" path.,
      2017-09-18)
      
      ---
      
      Another reproduction of the timestamp arithmetic problem
      
      select
         TIMESTAMP('2021-01-02 00:00:00') - TIMESTAMP('2021-01-01 00:00:00') a,
         20210102000000 - 20210101000000 aa,
         TIMESTAMP('2021-06-01 00:00:00') - TIMESTAMP('2021-05-31 00:00:00') b,
         20210601000000 - 20210531000000 bb,
         TIMESTAMPDIFF(second, TIMESTAMP('2021-05-31 00:00:00'), TIMESTAMP('2021-06-01 00:00:00')) c
      
      | a | aa | b | bb | c |
      | ------ | ------ | ------ | ------ | ------ |
      | 1000000 | 1000000 | 70000000 | 70000000 | 86400 |
      e83e7960
    • Jérome Perrin's avatar
      9a518828
    • Jérome Perrin's avatar
      tests/SecurityTestCase: sort groups in assertion failure message · f1e4e1c3
      Jérome Perrin authored
      This is a bit easier to read
      f1e4e1c3
    • Jérome Perrin's avatar
      74157942
    • Jérome Perrin's avatar
      Zelenium: catch potential errors when making (HTML) link for failed test · bb7edb27
      Jérome Perrin authored
      When a test fail, we make a data-url link with the HTML of the current
      page, so that we can easily investigate test failures n test nodes.
      We should not let errors that might happen here propagate, otherwise
      the test result is not created and the test runner does not detect
      that the test is finished.
      
      One case that caused such errors was failed assertion just after
      using goBack command without waiting
      bb7edb27
  3. 08 Jun, 2022 2 commits
  4. 06 Jun, 2022 6 commits
  5. 01 Jun, 2022 1 commit
  6. 27 May, 2022 1 commit
    • Jérome Perrin's avatar
      AlarmTool: handle automatic solve with alarms owned by system user · 58f6b8dc
      Jérome Perrin authored
      Business templates are installed by system user, which is a special
      user not returned by getWrappedOwner. Because of this, the "fixing
      problems or activating a disabled alarm is not allowed" error was
      raised when checking if the owner of the alarm has manage portal
      permission on the alarm.
      
      This switches the implementation to explicit creation of the user
      when user id is the system user, so that we have a user with the
      permission to solve the alarm.
      58f6b8dc
  7. 26 May, 2022 5 commits
  8. 24 May, 2022 6 commits
    • Valentin Benozillo's avatar
      erp5_ui_test: Drop the plural · d17614c7
      Valentin Benozillo authored
      d17614c7
    • Valentin Benozillo's avatar
      ui_test: Add range browser error test · 3998056d
      Valentin Benozillo authored
      When start and end are setup on float/integer field, the max and mic attributes are also setup in the input html element.
      So if the user enter a number outside this range, the browser should display an error.
      3998056d
    • Valentin Benozillo's avatar
      erp5_ui_test: Add external validator for float field range check · 789da4e7
      Valentin Benozillo authored
      using start and end filed on float, trigger web-browser error. But UI test are looking for multiple fields check by erp5.
      2 validators script are added because no argument can be passed to external validator, so :
      TALES: python: context.Base_ValidateFloatRange(value, mix=1, max=2)
      doesn't work
      789da4e7
    • Jérome Perrin's avatar
      *: keep using func_code and not yet __code__ with scripts · eaae74a0
      Jérome Perrin authored
      On Zope2, python scripts do not have __code__, they only have
      func_code (and same for __defauls__/func_defaults).
      We tried to backport the support of __code__ from Zope4 as a Zope2
      patch - it was SlapOS patch 4fa33dfc6 (erp5: py3: `func_{code,defaults}`
      was replaced in Python3 by `__{code,defaults}__`., 2022-04-25),
      but this patch was incomplete. We tried to backport more, but then
      realized that we don't need to use __code__ on ERP5 master yet,
      because ERP5 master branch is still supporting Zope2 only.
      
      This patch revert a small part of a17bb910 (py2/py3: Make Products
      code compatible with both python2 and python3., 2022-04-13), the part
      where we use f.__code__ where f might be a python script. For now,
      we'll apply this patch only on the Zope4 branch.
      
      A few places where f.func_code was used and f was a for sure not a
      python script but a simple class method or function are kept here, as
      __code__ support is missing only on in ZODB scripts.
      eaae74a0
    • Jérome Perrin's avatar
      tests: infrastructure to rebuild and export all business templates · 83e69b6b
      Jérome Perrin authored
      This is not really a test, but it reuses runUnitTest/runTestSuite
      commands, because they are good tools to quickly create ERP5
      environment and installing business templates.
      
      To re-build and re-export all* business templates, use this command:
      
          ./bin/runTestSuite --test_suite=ReExportERP5BusinessTemplateTestSuite
      
      --node_quantity argument can also be used to process multiple
      business templates in parallel.
      
      * note that this does not actually handle all business templates, but
      only the ones for which coding style test is enabled, because most
      business templates for which it is not enabled can not be installed.
      
      This typically produces large diffs that should apply the same
      change to many files and ideally, nothing else. We also developed a
      simple tool which summarize the diff by detecting the same chunk
      present in multiple files, it can be found at
      https://lab.nexedi.com/nexedi/erp5/snippets/1171 and also below.
      
      ---
      
      from __future__ import print_function
      """report similar hunks in a patch.
      """
      __version__ = '0.1'
      
      import argparse
      import collections
      import codecs
      import unidiff # unidiff==0.7.3
      import hashlib
      
      parser = argparse.ArgumentParser()
      parser.add_argument('patch_file', type=argparse.FileType('r'), default='-', nargs='?')
      parser.add_argument('-v', '--verbose', action='count', default=0)
      args = parser.parse_args()
      
      patchset = unidiff.PatchSet(codecs.getreader('utf-8')(args.patch_file))
      
      chunks_by_filenames = collections.defaultdict(set)
      
      for patch in patchset:
        for chunk in patch:
          chunk_text = u''.join([unicode(l) for l in chunk])
          chunks_by_filenames[chunk_text].add(patch.path)
      
      for chunk_text, filenames in chunks_by_filenames.items():
        chunk_hash = hashlib.md5(chunk_text.encode('utf-8')).hexdigest()
        print("Chunk %s is present in %s files" % (chunk_hash, len(filenames)))
        if args.verbose:
          print()
          print("\n".join("  " + f for f in sorted(filenames)))
          print()
        if args.verbose > 1:
          print()
          print(chunk_text)
          print()
      83e69b6b
    • Jérome Perrin's avatar
      accounting: always compare rounded values when indexing balance transactions · b075c99b
      Jérome Perrin authored
      To prevent rounding errors, we always compare rounded values to the
      precision of the accounting currency. There was a place here where we
      were using -= without rounding, which caused to detect a difference
      between new and current stock and insert a line for 0 in an existing
      instance for which some accounting lines were created with too precise
      values - but not in a way that was detected by the assertions in
      AccountingPeriod_createBalanceTransaction.
      
      Rounding here as well solved the problem with that data.
      b075c99b
  9. 23 May, 2022 2 commits
  10. 21 May, 2022 1 commit
    • Jérome Perrin's avatar
      ERP5Type/patches: prepare for removal of Products.DCWorkflowGraph · 88321109
      Jérome Perrin authored
      Supports the case where Products.DCWorkflowGraph is not present.
      Even though we are removing Products.DCWorkflowGraph from the
      software release, we don't remove this monkey patch yet, because
      this monkey patch also fixed a severe security issue. We keep the
      patch for the cases where a recent ERP5 runs on an old SlapOS where
      the product is still there.
      
      This change just moves the existing code in a try/except ImportError
      block
      88321109
  11. 19 May, 2022 1 commit
  12. 18 May, 2022 4 commits
    • Jérome Perrin's avatar
      85317472
    • Levin Zimmermann's avatar
      restricted: Allow patched pandas.read_* functions · 4360dbc6
      Levin Zimmermann authored
      Rationale:
      
      Converting * to data frame / numpy array efficiently is required in all
      wendelin projects, without this functionality wendelin is useless.
      Currently all projects allow this functionality in an insecure way.
      This commit aims to improve the situation by supporting a secure way of
      this functionality.
      
      (See wendelin!99 (comment 158474))
      
      Because pandas (in restricted Python) can also be useful in 'pure' ERP5
      (without Wendelin) the functionality is added to ERP5 source code.
      
      ---
      
      Security:
      
      Security is guaranteed by patching selected read_* functions and
      allowing the patched versions. The patch prohibits anything but
      string input which directly contains the data (e.g. no urls, file
      paths). New unit tests ensure the restrictions of the patches
      are actually effective.
      
      ---
      
      Notes on implementation decisions:
      
      Instead of offering new ERP5 extension methods (e.g. Base_readJson)
      this commit adds patched pandas read functions in restricted Python.
      In this way the change of the known API is as minimal as possible.
      
      Instead of globally monkey-patching pandas read_* functions, only the
      functions inside restricted python are patched.
      In this way the fully-functional, original functions are still available
      in Zope products or ERP5 extension code.
      
      Minor changes in the way how pandas is allowed in restricted python
      have been applied. Please consult the following discussions in the Merge
      request for details:
      
      !1615 (comment 159203)
      !1615 (comment 159341)
      4360dbc6
    • Levin Zimmermann's avatar
      add data configuration portal type group for wendelin · 70b92437
      Levin Zimmermann authored
      See merge request !1630
      70b92437
    • Levin Zimmermann's avatar
      ERP5Site: Remove compatibility with old data model · b1bdb286
      Levin Zimmermann authored
      ...for  getPortalDataConfigurationTypeList.
      
      See !1630 (comment 159889).
      b1bdb286