These exist in debian 11's /etc/mimetypes but not on debian 10. This
was causing testOooHandler failures on debian 10

See merge request !32

( tested with 7.5.2.2 )

timing seems to be slightly different (I did not investigate much)

This was probably for python2.6 compatibility, nowadays we can use
standard library's unittest

We want the test to test the actual code being ran as much as
possible, patching here seems not needed anymore.

Log tracebacks on error

lazy loging and other obvious changes

This does not need the types module which no longer exist

This test tried tries to convert a broken file, but this broken file was
"not broken enough" and it managed to pass and reply with 200 OK.

Adjust the test to use a "more broken file" that actually cause a 402
return code.

This is because with libreoffice 7.5.2.2 that file was broken enough
to trigger the error, but changing here minimize the changes related to
libreoffice version up.

On first soffice start with an empty profile, the process
immediatly exits with returncode 81 - this is confirmed in
https://bugs.documentfoundation.org/show_bug.cgi?id=107912 and
not really a bug. This bug suggest running with soffice, but
this seems not really needed for us, because we expect that
libreoffice may crash and we restart it in that case.

Problem is that util.waitStartDaemon(process) does not check
if process is still running, so if the process terminates we
have to wait for the timeout.

This changes waitStartDaemon to check for terminated process
in every iteration and to stop waiting when process is
terminated.

waitStopDaemon is also changed to make use of the new hasExited
method for consistency.

this will be compatible with python3

using 2to3 -f asserts -f except and a few manual
changes

Because LibreOffice does not offer a safe langage (e.g. something
without the Shell funtion), let's go with Python and disable this
feature by default.

For example, it can be used to force the type of a column
when converting from a CSV document.

A lot of changes because the import filter depends on the destination format.
This is equivalent to specifying --calc when using the command-line API.

See merge request !31

instead of re-implementing our own. Updating to zope.interface 5.4
caused test failures:

      File "cloudooo/handler/ooo/tests/testOooInterface.py", line 90, in testIOdfDocument
        self.assertEquals(IOdfDocument.names(), method_list)
    AssertionError: Lists differ: ['getFile', 'source_format', '... != ['getContentXml', 'parsed_cont...

    First differing element 0:
    'getFile'
    'getContentXml'

    - ['getFile', 'source_format', 'getContentXml', 'parsed_content']
    + ['getContentXml', 'parsed_content', 'source_format', 'getFile']

This is because Interface.name() order is unspecified.

As we could see in the XXX comments in the test, hardcoding the method
names from the interface was not ideal, using the built-in verifyClass
is better because it performs more checks and because error reporting
is also much more detailed.

cf. https://zopeinterface.readthedocs.io/en/latest/verify.html

Over the years (for example commits 89d754f2, 58a72658, a6fc7dfe, etc),
the signature of the implementation loadSettings evolved and the
interface became outdated.

This updates the interface with current implementation, so that we can
use zope.interface.verify.verifyClass in the tests

cloudooo includes its own mimes.types, so that it does not depend on
system configuration and behave the same regardless of the underlying
system, but the embedded mime.types have to be loaded explicitly by
calling utils.loadMimetypeList() by each python process before using
mimetypes module.

This was not done for ooo and x2t handlers, so in practice they were
depending on the system mime.types and the tests were written to expect
the mimetypes from debian 10, but with debian 11 some mimetypes became
different ( for example .bmp extension was guessed as image/x-ms-bmp [1]
on debian 10 and image/bmp on debian 11 [2]),

Theses changes:
 - include mime.types from debian 10 [3], but keeping the extra mimetypes
   for  only-office documents (docy, ppty and xlsy) that were added in
   0bb5fbdc (x2t: add handler, 2016-09-22)
 - change the handlers to call utils.loadMimetypeList(), it was only
   strictly necessary for ooo handler, but do it as well in x2t for
   consistency.
 - adjust some tests in testX2tHandler, because now that we have
   loaded mimetypes database the mimetype of xlsy is returned in
   metadata, so we have "application/x-asc-spreadsheet" and not just "xlsy"
 - add a few more test for xlsy and docy, because the magic based test
   just verify that they are zip files, these new tests also make a few
   assertions on the content of the zip files

[1]: https://salsa.debian.org/debian/mime-support/-/blob/debian/3.62/mime.types#L677
[2]: https://salsa.debian.org/debian/media-types/-/blob/4.0.0/mime.types#L1804
[3]: https://salsa.debian.org/debian/mime-support/-/blob/debian/3.62/mime.types

zope.interface does not use self argument.

Also set a pylint directive not to report false positive regarding
the lack of self argument.

This uses psutil.process_iter to iterate on all processes, but in the meantime
some process might terminate.

As we could observe in the logs:

    2021-04-26 12:53:02 - Cloudooo - DEBUG - Stop Pid - 15816
    Exception in thread Thread-1:
    Traceback (most recent call last):
      File "python2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "cloudooo-repository/cloudooo/handler/ooo/monitor/request.py", line 60, in run
        self.openoffice.restart()
      File "cloudooo-repository/cloudooo/handler/ooo/application/application.py", line 79, in restart
        self.stop()
      File "cloudooo-repository/cloudooo/handler/ooo/application/openoffice.py", line 169, in stop
        self._releaseOpenOfficePort()
      File "cloudooo-repository/cloudooo/handler/ooo/application/openoffice.py", line 122, in _releaseOpenOfficePort
        if process.exe() == join(self.office_binary_path, self._bin_soffice):
      File "psutil-5.8.0-py2.7-linux-x86_64.egg/psutil/__init__.py", line 660, in exe
        exe = self._proc.exe()
      File "psutil-5.8.0-py2.7-linux-x86_64.egg/psutil/_pslinux.py", line 1688, in exe
        raise NoSuchProcess(self.pid, self._name)
    NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=19382)

Update to catch and ignore such exceptions, and minor refactorings such as:
 - Not calculate the path of openoffice binary in the loop, this is not
   supposed to change.
 - Remove reference to lsof in logged message, this does not use lsof.
 - Catch all exceptions.

In python2, mimetypes module was not deterministic and this snippet:

    import mimetypes; print(mimetypes.guess_extension("text/html"))

use to be ".html" on python 2.7.14, but is ".htm" on python 2.7.18

Similarly:

    import mimetypes; print(mimetypes.guess_extension("application/msword"))

was ".doc" on 2.7.14 and ".dot" on 2.7.18 (this was in my observations, this
does not look deterministic as it is iterating on a dict, maybe this behaviour
is not always same)

For html conversion engine, ERP5 expect that the extension for text/html is
.html, not .htm. Some tests also expect that the conversion for word is .doc so
to keep compatibility with extensions used in ERP5 compatibility mode, define
explicitly these two extensions instead of depending on python standard library.

See merge request !25

See merge request !24