Revert "erp5_calendar: Update stop_date, otherwise it can stay in the past and generate wrong duration (stop_date - start_date)"

This reverts commit 6af8eca1f4d7fbdd29cd001b1c5a7d4d03b51990.

SimulationTool_zGetInterpolationMethod is called in restricted context
of Resource_zGetMovementHistoryList so it needs to be usable by
anonymous.

ZSQL Method does not read parameters from REQUEST, so it looks safe.

erp5_calendar: Update stop_date, otherwise it can stay in the past and generate wrong duration (stop_date - start_date)

With this, we accept users with permission to Modify portal content or Associate role

With this, change we accept users with permission to Modify portal content or Associate role

jerome/erp5!6 rebased and in a big
commit so that we can start using that
SimulationTool: implement "linear" flow API

Conflicts:
	product/ERP5/bootstrap/erp5_core/SkinTemplateItem/portal_skins/erp5_core/Resource_zGetInventoryList.xml

testInventoryAPI: silent pyflakes warnings

more interpolation implementation

inventory_list interpolation flow: support at_date & to_date

more and more interpolation API

interpolation

interpolation

group_by_time_sequence_list

testInventoryAPI: move inventory valuation methods in a dedicated test class

This should not be in test inventory list.

test: move Tracking API to a dedicated test module

inventory api: dirty way of disabling cache if keys such as node_category are used.

interpolation: security on SimulationTool_zGetInterpolationMethod

We should be able to configure a group calendar saying that the pattern
is "from 9:00 to 12:00, repeat every monday morning" with a group
calendar assignment saying "use this pattern from 01/01/2016 until
31/12/2016" and then create another group calendar assignment for 2017
without having to change the periodicity stop date on all presence
periods of the group calendar.

I think it should repeat from group calendar assignment's start date
until min(group calendar assignment's stop date, presence period's
periodicity stop date).

(cherry picked from commit b42a9c112d59454a42c0b5a0fd62d71a7fcd4202)

 - update PresencePeriod.getNextPeriodicalDate with fixes from 6155f7ff
 - do not use addToDate, but simply DateTime arithmetics that unlike addToDate, works correctly

(cherry picked from commit 30e2c1f6cb55ae3dfc4d8c83911b53f38a6cc36d)

because Domain.getRelativeUrl (and Category.getRelativeUrl) cannot be
restrictedTraverse'd

Workaround for bug #20160120-1805D22

The real motivation was to make sure that search match is displayed when searching with Ctrl+F / Ctrl+G

Conflicts:
	bt5/erp5_code_mirror/SkinTemplateItem/portal_skins/erp5_code_mirror/code_mirror_support.xml

codemirror: display an animation in window title so that the window flash when component has been saved.

Use case: save component on one tab, go to "run live test" tab and wait for component to be saved to make sure code runs with latest version

Conflicts:
	bt5/erp5_code_mirror/SkinTemplateItem/portal_skins/erp5_code_mirror/code_mirror_support.xml

This is probably incorrect. I believe ERP5ShortMessage's message_type
handling has to be refactored. I guess this should not be part of the
send API but be a property set on the gateway instance in portal_sms

Only mobyt support this parameter, so this break the compatibility with
essendex

Payroll related document classes (AnnotationLine,  PaySheetLine, PaySheetTransactionLine, PaySheetCell) are not different from Invoice-related classes.

This work aims to remove these document classes and base the portal types on the generic invoice classes.

This work started after finding out that Pay Sheet Lines were appearing in the "Account Statement" report, on side of Pay Sheet Transaction Lines. This happened as PaySheetLines is overiding the "isAccountable" method to return True. This merge request, besides removing duplicate code, fixes the accountable property of Pay Sheet Lines, and updates the call to the inventory API to generate correct accounting/payroll reports.

/reviewed-on nexedi/erp5!223

93e30e5e introduce a  new `user` table that is listed as a *search table*, so until this table is created all catalog queries are failing, and alarm tool, which relies on catalog fail with such a traceback:

```
ERROR TimerService Process timer error
Traceback (most recent call last):
  File "parts/erp5/product/TimerService/TimerService.py", line 102, in process_timer
    DateTime(prev_tick), DateTime(next_tick))
  File "parts/erp5/product/ERP5/Tool/AlarmTool.py", line 175, in process_timer
    self.tic()
  File "parts/erp5/product/ERP5/Tool/AlarmTool.py", line 135, in tic
    for alarm in self.getAlarmList(to_active=1):
  File "parts/erp5/product/ERP5/Tool/AlarmTool.py", line 111, in getAlarmList
    alarm_date={'query':now,'range':'ngt'}
  File "parts/erp5/product/ERP5Catalog/CatalogTool.py", line 702, in unrestrictedSearchResults
    return ZCatalog.searchResults(self, **kw)
  File "parts/erp5/product/ZSQLCatalog/ZSQLCatalog.py", line 1091, in searchResults
    return catalog.searchResults(REQUEST, **kw)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2585, in searchResults
    **kw
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2554, in queryResults
    **kw
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2418, in buildSQLQuery
    ignore_unknown_columns=ignore_unknown_columns,
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2394, in buildEntireQuery
    query=self.buildQuery(kw, ignore_empty_string=ignore_empty_string, ignore_unknown_columns=ignore_unknown_columns),
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2295, in buildQuery
    result = self.buildSingleQuery(key, value)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2087, in buildSingleQuery
    search_key, related_key_definition = self.getColumnSearchKey(key, search_key_name)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 2049, in getColumnSearchKey
    related_key_definition = self.getRelatedKeyDefinition(key)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 1999, in getRelatedKeyDefinition
    for entire_definition in self.getSQLCatalogRelatedKeyList([key]):
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 1935, in getSQLCatalogRelatedKeyList
    column_map = self._getSQLCatalogRelatedKeySet()
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 130, in wrapper
    result = transactional_cache[cache_id] = method(wrapped_self)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 1908, in _getSQLCatalogRelatedKeySet
    column_map = self.getColumnMap()
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 130, in wrapper
    result = transactional_cache[cache_id] = method(wrapped_self)
  File "parts/erp5/product/ZSQLCatalog/SQLCatalog.py", line 1146, in getColumnMap
    for field in table_dict[table]:
KeyError: 'user'
```

This means that even if upgrader's post upgrade constraint are supposed to fix this by calling `portal_catalog.upgradeSchema`, because this constraint rely on alarm tool, it's already too late.

The suggested way to fix this is to also call [`portal_catalog.upgradeSchema`](https://lab.nexedi.com/nexedi/erp5/blob/c99fb9163503c5afdef59ecb124b3060b05330a4/bt5/erp5_upgrader/SkinTemplateItem/portal_skins/erp5_upgrader/TemplateTool_checkTableConsistency.py#L7) in the same transaction that the transaction upgrading business templates.

I have not completely removed `TemplateToolTableConsistencyConstraint`, because maybe there will be cases where just call post upgrade alarm to execute data migration steps after installing a business template manually. But you guys think it's better to completely merge `TemplateToolTableConsistencyConstraint` with `TemplateToolBusinessTemplateInstallationConstraint` I would be OK with that too.

Of course, we cannot just make
`TemplateToolTableConsistencyConstraint` an `upgrade` constraint, because there would be no guarantee that it's called after `TemplateToolBusinessTemplateInstallationConstraint`.

Also, I have not considered making [`getColumnMap` and friends](https://lab.nexedi.com/nexedi/erp5/blob/master/product/ZSQLCatalog/SQLCatalog.py#L1142) resilient to the case where a table listed in search tables does not exist, because it's would just have been failing later anyway and it's better to fail early in such case.

cc: @vpelletier @jm @seb @tiwariayush @gabriel

/reviewed-on nexedi/erp5!247

We had an incident in a in instance were a user changed state of 70K invoices using https://www.erp5.com/howto/erp5-developer-howto/erp5-HowTo.Change.Workflow.State.Of.Multiple.Documents  and leads to user not receiving the reports they requested.

`Folder_modifyWorkflowStatus` creates a lot of `callMethodOnObjectList` activities (for all the selected documents) with priority 2, then these activities will cause more reindex an expand activities.

Until all these `callMethodOnObjectList` are processed, no new activities with priority > 2 were processed. Some "important for users" activities such as erp5_deferred_style reports where waiting in the queue.

I believe we should just set a lower priority to these `callMethodOnObjectList`, eventhough I considered a more clever way of giving the priority:

 - if the number of selected documents is reasonably small, process them with a very high priority, this way the user can see his document changing state almost immediately as when using the synchronous change state. This should not cause congestion because there are not too many documents.
 - when there are a lot of selected documents, process them with a very low priority, because anyway it will take time and user will not "wait" for documents to change state.

... but I realized this is trying to be too clever.

So any objections to just lower priority here ?

/reviewed-on nexedi/erp5!235

As discussed in  nexedi/erp5!248 this approach allows to set any mail header.

I also included a not so related patch of email header handlings 88d40b40 so that we review all this together.

/cc @gabriel @kazuhiko 

/reviewed-on nexedi/erp5!256

I just simulated an upgrade for an ERP5 running a version from before !185,  `ERP5UserManager Non Existence_constraint` properly disabled old `acl_users/erp5_users` and migrated all persons, but  `ERP5UserLoginManager Existence_constraint` did not activate `acl_users/erp5_login_users`.

This is because  `ERP5UserLoginManager Existence_constraint` is registered as a **pre-upgrade** constraint, but with my upgrade scenario, pre-upgrade constraint are installed too late. 

The upgrade scenario was:

 1. install new ERP5 SR on slapos
 2. manually install new versions of  erp5_upgrader and customer_configuration_upgrader business template
 3. run upgrade ( sense then fix on portal_alarms/promise_check_upgrade )

During step 3, when pre-upgrade constraints are executed, erp5_base is not installed yet, so  `ERP5UserLoginManager Existence_constraint` can not run because it's not installed yet. 

My suggestion is to change it to a post-upgrade constraint.

If I understand correctly and my upgrade scenario is really the supported one, it means we can only have pre-upgrade constraints in erp5_upgrader.

( I also fix a meaningless typo in the same code)

/cc @kazuhiko  @vpelletier @seb @gabriel 

/reviewed-on nexedi/erp5!262

To be able to use pre_upgrade we would need to install the bt first

As Pay Sheet Lines, Pay Sheet Transaction Lines and Annotation Lines
have been made generic using the Invoice Line class

As Pay Sheet Line has now Invoice Line as base class

Because of ERP5Type.UnrestrictedMethod, 'System Processes' can own objects.
Such objects can be proxy-role'd scripts, and proxy-role mechanism
triggers many users look-ups (each time security is evaluated, which is
virtually every getattr). Each such lookup will do a query for 'System
Processes' user, which will (hopefully) find nothing anyway.
So special-case 'System Processes' when looking by user_id by skipping
the search altogether (enforcing the inability to locate this user,
consistently with Zope assumptions, and consistently with previous
behaviour).

The code was not working as expected when we had several test nodes with same name
(which should not happen). So add a timestamp in the title.

test_result: fixed random issues where test result was staying confirmed while all tests were finished

When two test results line were stopped in parallel, both transactions were thinking that
there was still one ongoing test line, thus none of them were setting the test result in stopped.

Use activities instead of serialization to avoid having testnode getting conflict errors.