1. 18 Jun, 2021 1 commit
    • Kirill Smelkov's avatar
      runUnitTest: Close ZODB DB before shutdown · 171a66f2
      Kirill Smelkov authored
      We are already closing Storage on shutdown (see nearby Storage.close call in
      the patch), but the DB handle was not closed. With classic ZODB it does not
      really matter in practice, because not closing DB is only leaking RAM resources
      and the program is anyway terminated soon.
      
      However with wendelin.core 2 things are different: in addition to ZODB storage
      server, there is also synthetic WCFS filesystem from which files are opened
      and memory-mmaped. In runUnitTest we start both ZODB and WCFS servers and we
      also shut down them both in the end. The filesystem server can be cleanly
      unmounted and shutdown only when there are no opened files left on it.
      
      Wendelin.core 2 client works by complementing each ZODB connection (zconn) with
      WCFS-level connection (wconn) to WCFS server. Those two zconn and wconn are
      kept in sync by wendelin.core 2 client logic: whenever zconn adjusts its view
      of the database, so does wconn. And whenever zconn is garbage collected, so
      does wconn is closed to release resources and close corresponding files opened
      on WCFS. In addition to garbage-collection, wconn is also closed when zconn.db
      - the ZODB DB handle via which zconn was created - is closed. This is needed to
      be able to reliably trigger freeing WCFS resources, because even after DB is
      closed, zconn can still stay alive forever being referenced from some python
      object - e.g. a frame or traceback or something else.
      
      The latter scenario actually happens during runUnitTest run. As the result it
      leads to inability to unmount and stop WCFS server cleanly:
      
          $ ./runUnitTest --load --save --with_wendelin_core -v erp5_core_test:testWendelinCore
          ...
          test (erp5.component.test.erp5_version.testWendelinCore.TestWendelinCoreBasic) ... ok
      
          ----------------------------------------------------------------------
          Ran 1 test in 0.105s
      
          OK
          F0618 19:05:46.359140   35468 wcfs/client/wcfs.cpp:486] CRITICAL: pinner: wcfs /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2: wlink25: recvReq: link is down
          F0618 19:05:46.359173   35468 wcfs/client/wcfs.cpp:487] CRITICAL: wcfs server will likely kill us soon.
          CRITICAL: pinner: wcfs /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2: wlink25: recvReq: link is down
          CRITICAL: wcfs server will likely kill us soon.
          Traceback (most recent call last):
            File ".../bin/runUnitTest", line 312, in <module>
              sys.exit(runUnitTest.main())
            File ".../parts/erp5/Products/ERP5Type/tests/runUnitTest.py", line 926, in main
              run_only=run_only,
            File ".../parts/erp5/Products/ERP5Type/tests/runUnitTest.py", line 709, in runUnitTestList
              wcfs_server.stop()
            ...
            File ".../parts/wendelin.core/wcfs/__init__.py", line 543, in _fuse_unmount
              raise RuntimeError("%s\n(more details logged)" % emsg)
          RuntimeError: fuse_unmount /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2: failed: fusermount: failed to unmount /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2: Device or resource busy
          (more details logged)
      
          # logs
          2021-06-18 19:05:45.978 INFO root wcfs: unmount/stop wcfs pid32981 @ /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2
          2021-06-18 19:05:46.068 WARNING root fuse_unmount /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2: failed: fusermount: failed to unmount /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2: Device or resource busy
          2021-06-18 19:05:46.068 WARNING root # lsof /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2
          2021-06-18 19:05:46.357 WARNING root COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
          runUnitTe 32175 slapuser34   24r   REG   0,48      111    4 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/.wcfs/zurl
          runUnitTe 32175 slapuser34   25u   REG   0,48        0    7 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/head/watch
          runUnitTe 32175 slapuser34   26r   REG   0,48  2097152    9 /dev/shm/wcfs/b53b61099c740b452b383db6df6dce4ad6d23ba2/head/bigfile/00000000000078b4
      
          2021-06-18 19:05:46.358 WARNING root -> kill -TERM wcfs.go ...
          2021-06-18 19:05:46.358 WARNING root -> abort FUSE connection ...
      
      I've debugged things a bit and even with
      kirr/ZODB@bbd03b3a ZODB connection stays alive
      being referenced from some frame objects.
      
      -> Fix this problem by explicitly closing ZODB DB nearby tests shutdown before call to wcfs_server.stop
      171a66f2
  2. 17 Jun, 2021 1 commit
    • Kirill Smelkov's avatar
      tests: Teach test driver to pass testWendelinCore when run with wendelin.core 2 · 530e8b4e
      Kirill Smelkov authored
      This is follow-up to 5796a17a (core_test: Add test to make sure that
      wendelin.core basically works; nexedi/erp5!1429).
      
      In that commit it was said that testWendelinCore
      
          "currently passes with wendelin.core 1, which is the default.
           It also passes as live test with wendelin.core 2.
           However with wendelin.core 2 it currently fails when run on testnodes
           ...
           because we need to amend ERP5 test driver
      
           1. to run tests on a real storage instead of in-RAM Mapping Storage(*), and
           2. to spawn WCFS server for each such storage."
      
      This patch addresses that latter problem to run testWendelinCore under
      testnode infrastructure.
      
      @rafael and @jerome suggested that we can force a test to be run on a
      real storage via `runUnitTest --load --save` or via `--activity_node=n`.
      
      @rafael also suggested not to generally change the testing driver, but instead
      make step-by-step progress and first tag each test that uses wendelin.core with an
      option. Let's go this way now: runUnitTest/custom_zodb are taught to launch
      WCFS server if wendelin.core usage is requested and software is built with
      wendelin.core 2.
      
      With both changes combined testWendelinCore should now pass OK when run
      on a testnode with both wendelin.core 1 and wendelin.core 2.
      
      This patch is based on a draft patch by @rafael: https://lab.nexedi.com/rafael/erp5/commit/14e3a777.
      
      This patch also relies on recent wendelin.core 2 wcfs.py rework which
      exposed functionality to start WCFS server and to further control it:
      kirr/wendelin.core@5bfa8cf8.
      
      /cc @tomo, @romain, @jerome, @seb
      530e8b4e
  3. 15 Jun, 2021 3 commits
  4. 14 Jun, 2021 2 commits
  5. 10 Jun, 2021 1 commit
  6. 09 Jun, 2021 2 commits
    • Jérome Perrin's avatar
      core: keep using default_params from proxy listbox ids · afa5c0fc
      Jérome Perrin authored
      With bf57228a ([erp5_core] RelationField listbox must also fetch
      default_params value from the relationfield, 2018-05-09), we introduced a
      regression that listboxs from relation fields "proxy listbox ids" default params
      where no longer used. As a result, if we have a listbox which define some
      catalog search with default parameters, like for example the "Suppliers" or
      "Clients" listboxs from erp5_trade are doing, these parameters were not
      taken into account. In the case of trade listbox, this was showing all nodes.
      
      This changes Base_getRelatedObjectParameter to get catalog search from the
      proxy listbox if there is one.
      afa5c0fc
    • Jérome Perrin's avatar
      trade_renderjs_ui_test: tests for "Suppliers", "Clients", "Organisations" and... · f67f5346
      Jérome Perrin authored
      trade_renderjs_ui_test: tests for "Suppliers", "Clients", "Organisations" and "Persons" Proxy Listbox Ids
      f67f5346
  7. 08 Jun, 2021 5 commits
  8. 07 Jun, 2021 1 commit
  9. 04 Jun, 2021 7 commits
    • Jérome Perrin's avatar
      ERP5Type: fix logger "name" when logging warnings · e3804008
      Jérome Perrin authored
      Instead of using an always-different logger name which confuses log
      analytics system, use the warning category as logger name (or what zLOG.LOG
      calls "subsystem")
      e3804008
    • Jérome Perrin's avatar
      officejs: fix typo · 21c271cd
      Jérome Perrin authored
      21c271cd
    • Jérome Perrin's avatar
      trade/accounting: fix "Trade Condition" column on accounting module · d7bfd46d
      Jérome Perrin authored
      Rename this column from "Sale Trade Condition" to "Trade Condition", so
      that it can be used also for purchase and internal trade conditions.
      d7bfd46d
    • Jérome Perrin's avatar
      Payment mean and SEPA credit transfer · 66670bd2
      Jérome Perrin authored
       - update for ERP5JS compatibility
       - prevent an error with "Select Payments" update action, when limit was empty
       - export to generate SEPA credit transfer ( pain.001.001.02 )
       - new mode in "Select Payment" to select planned/confirmed payment and validate them at the same time
       - allow selecting lines individually in "Select Payments"
       - allow filtering by third party in "Select Payments" (only supported in xhtml_style for now)
       - fix issue that Deliver transaction leaves PTG in Delivering state 
       - french translations
      
      See merge request !1400
      66670bd2
    • Jérome Perrin's avatar
      corporate_identity_test: update expected images after updating ghostscript 9.54.0 · f6637a16
      Jérome Perrin authored
      Changes following up nexedi/slapos!985
      
      It seems the logo is a bit bigger, or at least a bit different
      
      See merge request nexedi/erp5!1424
      f6637a16
    • Jérome Perrin's avatar
      Lighter processing for OCR activities · 9e375b8e
      Jérome Perrin authored
      When running OCR, we sometimes have issues because processing is "too heavy":
       - [x] use 2 or 3 Go of disk space for a one page PDF created by erp5_document_scanner, because we convert pdf -> png -> tiff before sending to tesseract. Modern Ghostscript supports running tesseract directly, so we use it if it's available.
       - [x] use 300% of CPU. Fixed by setting `OMP_THREAD_LIMIT` when running tesseract. This will only apply when OCR from Images. OCR embedded in Ghostscript does not seem to need this.
       - [x] ... and often crash, so is restarted. This is fixed by updated tesseract.
      
      Updates of ghostscript and tesseract are part of nexedi/slapos!985
      
      See merge request nexedi/erp5!1420
      9e375b8e
    • Jérome Perrin's avatar
      xhtml_style: remove timestamp from ckeditor URLs · cde8c993
      Jérome Perrin authored
      This timestamp break officejs offline capabilities.
      
      This reaplies 40f6c8fe ([erp5_xhtml_style] ckeditor: drop hardcoded
      timestamp parameter in URL, 2017-10-16) on the updated CKEditor
      
      This was done with:
      
          find product/ERP5/bootstrap/erp5_xhtml_style/SkinTemplateItem/portal_skins/erp5_ckeditor/ckeditor/ -type f  | xargs sed -i 's/"L4KA"/""/g'
      
      L4KA was the timestamp for this build, that we can in the top of ckeditor.js
      cde8c993
  10. 03 Jun, 2021 13 commits
  11. 02 Jun, 2021 4 commits