1. 25 Feb, 2019 1 commit
  2. 21 Feb, 2019 2 commits
    • Julien Muchembled's avatar
    • Julien Muchembled's avatar
      CMFActivity: new activate() parameter to prefer executing on the same node · 301962ad
      Julien Muchembled authored
      The goal is to make better use of the ZODB Storage cache. It is common to do
      processing on a data set in several sequential transactions: in such case, by
      continuing execution of these messages on the same node, data is loaded from
      ZODB only once. Without this, and if there are many other messages to process,
      processing always continue on a random node, causing much more load from ZODB.
      
      To prevent nodes from having too much work to do, or too little compared to
      other nodes, this new parameter is only a hint for CMFActivity. It remains
      possible for a node to execute a message that was intended for another node.
      
      Before this commit, a processing node selects the first message(s) according to
      the following ordering:
      
        priority, date
      
      and now:
      
        priority, node_preference, date
      
      where node_preference is:
      
        -1 -> same node
         0 -> no preferred node
         1 -> another node
      
      The implementation is tricky for 2 reasons:
      - MariaDB can't order this way in a single simple query, so we have 1
        subquery for each case, potentially getting 3 times the wanted maximum of
        messages, then order/filter on the resulting union.
      - MariaDB also can't filter efficiently messages for other nodes, so the 3rd
        subquery returns messages for any node, potentially duplicating results from
        the first 2 subqueries. This works because they'll be ordered last.
        Unfortunately, this requires extra indices.
      
      In any case, message reservation must be very efficient, or MariaDB deadlocks
      quickly happen, and locking an activity table during reservation reduces
      parallelism too much.
      
      In addition to better cache efficiency, this new feature can be used as a
      workaround for a bug affecting serialiation_tag, causing IntegrityError when
      reindexing many new objects. If you have 2 recursive reindexations for both a
      document and one of its lines, and if you have so many messages than grouping
      is split between these 2 messages, then you end up with 2 nodes indexing the
      same line in parallel: for some tables, the pattern DELETE+INSERT conflicts
      since InnoDB does not take any lock when deleting a non-existent row.
      
      If you have many activities creating such documents, you can combine with
      grouping and appropriate priority to make sure that such pair of messages won't
      be executed on different nodes, except maybe at the end (when there's no
      document to create anymore; then activity reexecution may be enough).
      For example:
      
        from Products.CMFActivity.ActivityTool import getCurrentNode
        portal.setPlacelessDefaultReindexParameters(
          activate_kw={'node': 'same', 'priority': priority},
          group_id=getCurrentNode())
      
      where `priority` is the same as the activity containing the above code, which
      can also use grouping without increasing the probability of IntegrityError.
      301962ad
  3. 13 Feb, 2019 3 commits
  4. 05 Feb, 2019 8 commits
    • Julien Muchembled's avatar
      4b7acaa7
    • Julien Muchembled's avatar
      CMFActivity: remove old skin if any · afaa9d19
      Julien Muchembled authored
      afaa9d19
    • Julien Muchembled's avatar
      CMFActivity: drop DTML completely and use consecutive uids when possible · d64887cb
      Julien Muchembled authored
      This moves the remaining DTML queries to Python, dropping the 'activity' skin.
      
      Dealing with conflicts of uids is easier if the inserted uids are consecutive:
      now, only 1 random value is generated, as base uid. This also preserves the
      order of insertion, which is wanted for performance reasons:
      - No more random write in the primary index.
      - When modifying several lines of several documents, 1 document being processed
        at a time, we'd like that any grouped activity (usually indexation) follows
        the same order, so that a processing node prefer many lines from a few
        documents instead of mixing lines from too many documents at the same time.
        This is usually better for caches.
      d64887cb
    • Julien Muchembled's avatar
      CMFActivity: speed up console activity watcher · bcaf44a4
      Julien Muchembled authored
      Average age of activities is dropped because it would become
      too complicated to implement and it's useless information.
      bcaf44a4
    • Julien Muchembled's avatar
      CMFActivity: remove processing/processing_date columns and improve watchers · 499d8f8b
      Julien Muchembled authored
      The original goal was to improve performance by removing the
      `processing_node_processing` index and the queries that modified
      these 2 useless columns.
      499d8f8b
    • Julien Muchembled's avatar
      CMFActivity: do not refresh view too often when checking messages dependencies · a32c8a42
      Julien Muchembled authored
      The root call to getExecutableMessageList (i.e. the one from distribute)
      is fast enough and won't hold old revisions of the database for too long.
      It is also completely read-only so it won't lock anything.
      
      This caused useless communication with the server.
      a32c8a42
    • Julien Muchembled's avatar
    • Julien Muchembled's avatar
      CMFActivity: validate a message with a single SQL request per queue · 7e387bcb
      Julien Muchembled authored
      As shown in the following example, on a big catalog table,
      MariaDB is able to use several indices at the same time
      ('...' are obfuscated unique values):
      
        > analyze select SQL_NO_CACHE uid, relative_url from catalog where reference='...' OR relative_url='...';
        +------+-------------+---------+-------------+------------------------+------------------------+---------+------+------+--------+----------+------------+--------------------------------------------------+
        | id   | select_type | table   | type        | possible_keys          | key                    | key_len | ref  | rows | r_rows | filtered | r_filtered | Extra                                            |
        +------+-------------+---------+-------------+------------------------+------------------------+---------+------+------+--------+----------+------------+--------------------------------------------------+
        |    1 | SIMPLE      | catalog | index_merge | Reference,relative_url | Reference,relative_url | 768,767 | NULL |    2 |   2.00 |   100.00 |     100.00 | Using union(Reference,relative_url); Using where |
        +------+-------------+---------+-------------+------------------------+------------------------+---------+------+------+--------+----------+------------+--------------------------------------------------+
        1 row in set (0.00 sec)
      
      So mixing different dependency types with OR should be fine
      (no need to split into more subqueries and join with UNION).
      7e387bcb
  5. 18 Jan, 2019 3 commits
  6. 13 Jan, 2019 1 commit
    • Julien Muchembled's avatar
      Fix reindexing with custom grouping using 'group_id' · 707058d7
      Julien Muchembled authored
      This fixes:
      
        Traceback (innermost last):
          Module Products.CMFActivity.ActivityTool, line 1373, in invokeGroup
            traverse(method_id)(expanded_object_list)
          Module Products.ERP5Catalog.CatalogTool, line 946, in catalogObjectList
            super(CatalogTool, self).catalogObjectList(tmp_object_list, **m.kw)
          Module Products.ZSQLCatalog.ZSQLCatalog, line 813, in catalogObjectList
            **kw
        TypeError: catalogObjectList() got an unexpected keyword argument 'group_id'
      707058d7
  7. 08 Jan, 2019 3 commits
  8. 07 Jan, 2019 2 commits
    • Sebastien Robin's avatar
    • Julien Muchembled's avatar
      CMFActivity: better date ordering by using micro-precision · b82f3ba1
      Julien Muchembled authored
      Originally, uids somehow sorted messages by date of insertion, in particular
      for those that were created within the same second. But since random uids,
      such messages became validated or processed in random order.
      
      Note however that by default, messages created in the same transaction all have
      exactly the same date, so commit a42da4de
      ("CMFActivity: Do not use offset for scanning messages to validate.")
      forces us to keep the ordering on uids (in addition to priority/date).
      
      Existing instances will upgrade automatically, using the already existing code
      to upgrade tables in a generic way. You should see the following logs:
      
          INFO CMFActivity 'message_queue' table upgraded
          ALTER TABLE message_queue
            MODIFY COLUMN date datetime(6) NOT NULL AFTER uid,
            MODIFY COLUMN processing_date datetime(6) DEFAULT NULL AFTER processing
          INFO CMFActivity 'message_job' table upgraded
          ALTER TABLE message_job
            MODIFY COLUMN date datetime(6) NOT NULL AFTER uid,
            MODIFY COLUMN processing_date datetime(6) DEFAULT NULL AFTER processing
          INFO CMFActivity 'message' table upgraded
          ALTER TABLE message
            MODIFY COLUMN date datetime(6) NOT NULL AFTER uid,
            MODIFY COLUMN processing_date datetime(6) DEFAULT NULL AFTER processing
      
      
      /reviewed-on !820
      b82f3ba1
  9. 02 Jan, 2019 1 commit
  10. 31 Dec, 2018 1 commit
  11. 21 Dec, 2018 2 commits
  12. 20 Dec, 2018 2 commits
  13. 17 Dec, 2018 4 commits
  14. 14 Dec, 2018 2 commits
  15. 13 Dec, 2018 1 commit
    • Arnaud Fontaine's avatar
      Revert "mark file uploading tests as expected failure." as these Functional... · dd528bae
      Arnaud Fontaine authored
      Revert "mark file uploading tests as expected failure." as these Functional Tests are not supposed to fail.
      
      This reverts commit ade16831.
      
      enablePrivilege, and thus UniversalFileRead used to upload files, was disabled
      in Firefox 17. Since SlapOS Firefox has been upgraded, "The operation is insecure"
      error is raised when uploading files. However, this doesn't mean that these tests
      should be expected to fail (and if they are, they should be removed instead).
      dd528bae
  16. 12 Dec, 2018 1 commit
    • Jérome Perrin's avatar
      accounting: restrict restarting accounting period to Assignor · 366a760e
      Jérome Perrin authored
      This workflow involved an Assignee who can open, close, re-open and an
      Assignor who can close definitively.
      This is usually configured so that accountants are Assignee and CFO is
      Assignor.
      
      We realized that re-opening a Period that was previously closed is
      something we don't want the accountants to do without CFO's approval.
      
      To support this configuration, we only allow Assignor to re-open.
      
      Now Assignee can open and close temporarily and Assignor can re-open and
      close definitively.
      
      /reviewed-on nexedi/erp5!813
      366a760e
  17. 11 Dec, 2018 3 commits