/reviewed-on nexedi/erp5!965

ZODB Components: Add 'Origin' (source_reference) field showing from which FS module it was imported.

* pylint may return a false positive error which have to be disabled and
  failing to import it because of that requires to edit on the FS and try
  again so it is not practical for a whole Product. Instead it is easier
  to import it and not validate
* Validation was done only for 'Test Component' and 'Extension Component',
  but all imported Components had their consistency and source code checked
  and this is not consistent to not validate but do these checks.
* importFromFilesystem() was checking consistency and source code, and this
  was done again when validating.

So leave the imported ZODB Components as draft and let the developer fixes
issues upon validation before committing.

ZODB Components: List of migratable 'Module Components' in Products.XXX.* should include any kind of objects and not only modules.

This fixes Products.ERP5VCS.Git module not being displayed because
Products.ERP5VCS.__init__ only imports one of its class and not the whole
module (and the module was not imported anywhere else too) so it was not in
Products.ERP5VCS.__dict__.

useStdin=False => temporary file was not created.
useStdin=True  => temporary file was created.

When multiple repositories exists and two of them contain scalability_test package be wiser and really check if neeeded Test Suite class exists before trying to load it.

When running tests the path to git command is just git, because we have
set $PATH

updateConfiguration is a method to set the list of partitions, not to
add partitions to an existing computer. The previous (slapos.core
<1.5.0) implementation of slap proxy was incorrect and was adding more
partitions each time it was called, but that was never the indented
behavior.

Prepare the list of partitions and call updateConfiguration only once to
set all partitions.

This should fix SLAPOS-SR-TEST that are still using erp5.util

Some tests where patching SlapOSControler to replace methods by a no-op,
but this was never reset. I believe we should rewrite this using mock
which takes care of un-doing the patches for us and makes this testing
easier, but for now I'm just reseting the missing methods like all
others were already reseted.

configparser read options as strings, node_quantity is not supposed to
be an int.

pkg_resources.resource_string reads the file as binary, if we want to
substitute in this string we need to decode it.

ZODB Components: Monkey patch pylint to avoid 'No config file found, using default configuration' on stderr.

Another solution could have been to have a configuration file for pylint instead
of command line arguments but it depends on the type of Component (for example
'Interface Component' has specific arguments).

Value of text_content is an unicode string whose (inside XML) encoding is utf-8. This inconsistency raises in xml library thus it is safer to convert back to utf-8 which is the intended encoding anyway.

After upgrading bt5s, portal_sms may still ERP5BaseBroken until restarting Zope.

ZODB Components: Allow migration of {Interface,Mixin,Module,Tool} from Business Template UI and on all Products (not only Products.ERP5).

This introduces the following new ZODB Components:
  + Module Component: Non-Documents/non-persistent classes of modules usually
    found at the top-level of Products (eg Products.ERP5.XXX) on FS. Considering
    that all other Components types are actually Modules, make it the base class.
  + Tool Component: Tool directory of Products on FS (eg Products.ERP5.Tool.XXX).
    => DiffTool and CallableTool are now 'Tool Component' instead of plain
       'Document Component' and properly registered as Tools like FS Products Tool.

Skip CMFActivity and HBTreeFolder2 Products for now in migration View for now as
almost many Portal Type classes have ActiveObject or HBTreeFolder2 in their MRO
and these Products will be done at the end anyway...

This often divides the size by ~3,
and also memory usage if it is loaded again.

See nexedi/slapos.core@42b55b5f

Classes to hide the header or the footer of a Listbox in new render-js and the header in xhtml (since there we have nothing in the footer anyway)

/reviewed-on nexedi/erp5!947

/reviewed-on nexedi/erp5!934

- redirect to sync when app version changes
- fix configuration searchable fields in list form
- fix codemirror editor gadgetField
- fix exception catch in action script
- handle jio not found error for document

- document view
- upload and download

* Add erp5_core_proxy_field_legacy to dependencies (because of (at least) my_password).
* Rename ERP5ShortMessage.Errors module because the name is too generic for a 'Module Component'.
* Fix type_class of SMSTool: This should be the name of the class only, excluding the module.
* Add missing __init__.py to tests directory so that it can be migrated.

This is done by inheriting most of the code of ConflictFreeLog,
i.e. using a doubly-linked list:
- for fast iteration of the first elements, and in particular
  immediate access to the first element (used for creation date);
- that keeps track of the history length;
- that implement fast reverse iteration (although it could
  have been done without changing the data structure).

The size of buckets is not fixed anymore to 16 items:
like ConflictFreeLog, WorkflowHistoryList is also a good candidate
to look at the estimated serialized size of the bucket in order to
decide if elements should be added to a new one or not.
Then developers won't have to care about using Pdata or not.

The size is bigger than the ConflictFreeLog default,
because workflow items look a lot alike and adding
a few more is cheap when the ZODB compresses.

No more optimized __getstate__ (except for workflow histories that
have not been migrated) so BT export will be a bit more verbose.

The BBB code is because of
  !934

/reviewed-on !941

ZODB Components: Before migrating Interfaces from FS, there must be one Interface class per source file matching its name.

Same as Document class: this avoids registering Interfaces at startup and just do it
when using the Interface. In assuming that portal_components/XXX has a class
name equals to XXX.getRference(), it is easier to:
 * List all existing Interfaces (for example in Portal Type Class view): getReference()
   on all validated 'Interface Component' in portal_components.
 * Lookup for an 'Interface Component': 'from erp5.component.interfaces.XXX import XXX'.

ZODB Components: Properly handle addition of template_* properties in BusinessTemplate PropertySheet.

On the plus side, this avoids an ugly 'except AttributeError: pass' which does
not work anyway with the scenario below.

Assuming the following:
  * template_XXX Property (accessor: getTemplateXXX) recently added to BusinessTemplate PropertySheet.
  * erp5_YYY sets template_A Property (erp5_YYY/bt/template_XXX.xml).
  This commit handles the following:
    1. Stop instance with old ERP5 without template_XXX.
    2. Update erp5.git.
    3. Start instance.
    4. Upgrade erp5_property_sheets and erp5_YYY bt5s in *one* transaction from portal_templates UI.
    => When upgrading erp5_YYY, BusinessTemplate.importFile() imports erp5_YYY/bt/*.xml where the
       list of files is based on BusinessTemplate class propertyMap(), but at this point accessors
       have not been re-generated yet, thus template_XXX is not returned by propertyMap() and
       erp5_YYY/bt/template_XXX.xml is not imported.
       => portal_templates/erp5_YYY new BT does not have template_XXX property set at all.