-> Use .[test] to refer to them.
https://stackoverflow.com/a/41398850/9456786

/reviewed-by @jerome
/reviewed-on nexedi/zodbtools!14

@jerome, I was trying to make zodbtools work with Python3 and along that road picked some bits of your work from nexedi/zodbtools!12. At present the migration to Python3 is not complete, and even though now I have the answer to how handle strings in both python2/3 in compatible and reasonable way (I can share details if you are interested), I have to put that work on hold for some time and use https://pypi.org/project/pep3134 directly in wcfs tests, since getting all string details right, even after figuring on how to do it, will take time. Anyway the bits presented here should be ready for master and could be merged now. Could you please have a look?

Thanks beforehand,  
Kirill

/reviewed-on nexedi/zodbtools!13

Zodbdump format is mixed text+binary so dumping to unicode stdout won't
work.

Based on patch by Jérome Perrin.

Because on Py3:

        def test_dumpreader():
            in_ = b"""\
        txn 0123456789abcdef " "
        user "my name"
        description "o la-la..."
        extension "zzz123 def"
        obj 0000000000000001 delete
        obj 0000000000000002 from 0123456789abcdee
        obj 0000000000000003 54 adler32:01234567 -
        obj 0000000000000004 4 sha1:9865d483bc5a94f2e30056fc256ed3066af54d04
        ZZZZ
        obj 0000000000000005 9 crc32:52fdeac5
        ABC

        DEF!

        txn 0123456789abcdf0 " "
        user "author2"
        description "zzz"
        extension "qqq"

        """

            r = DumpReader(BytesIO(in_))
            t1 = r.readtxn()
            assert isinstance(t1, Transaction)
    >       assert t1.tid == '0123456789abcdef'.decode('hex')
    E       AttributeError: 'str' object has no attribute 'decode'

    test/test_dump.py:77: AttributeError

Based on patch by Jérome Perrin.

	self = <zodbtools.util.CRC32Hasher object at 0x7f887ae465f8>

	    def __init__(self):
	>       self._h = crc32('')
	E       TypeError: a bytes-like object is required, not 'str'

	util.py:208: TypeError

Based on patch by Jérome Perrin.

	data = 'data1'

	    def sha1(data):
	        m = hashlib.sha1()
	>       m.update(data)
	E       TypeError: Unicode-objects must be encoded before hashing

	zodbtools/util.py:38: TypeError

Based on patch by Jérome Perrin.

	s = b'\x03\xc4\x85v\x00\x00\x00\x00'

	    def ashex(s):
	>       return s.encode('hex')
	E       AttributeError: 'bytes' object has no attribute 'encode'

	zodbtools/util.py:29: AttributeError

s.encode('hex') used to work on Py2 but fails on Py3:

	In [1]: s = "abc"

	In [2]: b = b"def"

	In [3]: s.encode('hex')
	---------------------------------------------------------------------------
	LookupError                               Traceback (most recent call last)
	<ipython-input-3-75ae843597fe> in <module>()
	----> 1 s.encode('hex')

	LookupError: 'hex' is not a text encoding; use codecs.encode() to handle arbitrary codecs

	In [4]: b.encode('hex')
	---------------------------------------------------------------------------
	AttributeError                            Traceback (most recent call last)
	<ipython-input-4-ec2fccff20bc> in <module>()
	----> 1 b.encode('hex')

	AttributeError: 'bytes' object has no attribute 'encode'

	In [5]: import codecs

	In [6]: codecs.encode(b, 'hex')
	Out[6]: b'646566'

	In [7]: codecs.encode(s, 'hex')
	---------------------------------------------------------------------------
	TypeError                                 Traceback (most recent call last)
	/usr/lib/python3.7/encodings/hex_codec.py in hex_encode(input, errors)
	     14     assert errors == 'strict'
	---> 15     return (binascii.b2a_hex(input), len(input))
	     16

	TypeError: a bytes-like object is required, not 'str'

	The above exception was the direct cause of the following exception:

	TypeError                                 Traceback (most recent call last)
	<ipython-input-7-7fcb16cead4f> in <module>()
	----> 1 codecs.encode(s, 'hex')

	TypeError: encoding with 'hex' codec failed (TypeError: a bytes-like object is required, not 'str')

After the patch it works with bytes and raises for str.
Fromhex does not need to be changed - it already uses codecs.decode way as
originally added in dd959b28 (zodbdump += DumpReader - to read/parse zodbdump
stream).

Based on patch by Jérome Perrin.

There is no cStringIO on Python3:

	test_dump.py:26: in <module>
	    from cStringIO import StringIO
	E   ModuleNotFoundError: No module named 'cStringIO'

Based on patch by Jérome Perrin.

This makes zodb command driver tests added in the previous patch to pass
on both python2 and python3.

----

kirr: factor running `zodb ...` into zodbrun + add test for `zodb -h`.

Added test currently passes on py2, but fails on py3:

	out = <_io.TextIOWrapper encoding='UTF-8'>

	    def usage(out):
	        print("""\
	    Zodb is a tool for managing ZODB databases.

	    Usage:

	        zodb command [arguments]

	    The commands are:
	    """, file=out)

	        cmdv = command_dict.keys()
	>       cmdv.sort()
	E       AttributeError: 'dict_keys' object has no attribute 'sort'

	zodbtools/zodb.py:55: AttributeError

It will be fixed in the next patch.

----

kirr: use loggging as log and keep emitting warnings on one line.

this silents a warning about \w being unknown escape sequence

----

kirr: preserved _obj_re definition to be on 1 line.

until https://github.com/zopefoundation/ZODB/pull/183 gets merged, let's
run also the tests for this, since we have support for this extension.

also simplify a bit definition as ZODB is common in all versions

----

kirr:

- cover only last 2 py3 releases: 3.6 and 3.7 for now (3.8 is not yet
  released)
- separate ZODB3 as it supports only python2.

Py3 tests are failing for now and we'll be getting them to pass
incrementally - step by step.

Using dateparser to support absolute and relative dates in natural language.

/reviewed-by @kirr
/reviewed-on nexedi/zodbtools!8

Fixes to get all tests passing with all ZODB versions we care about.  

Organize testing coverage for all cases via tox, similarly to wendelin.core .

/reviewed-on nexedi/zodbtools!11

Use tox to test with all kinds of ZODB.
With preceding 3 patches tests pass with all versions of upstream ZODB.

TODO: test coverage for both py2 and py3.

maxtid is in ZODB.utils starting only from ZODB5.

ZODB{3,4} want txn._extension, while ZODB5 deprecate it in favour of
txn.extension.

IStorageTransactionMetaData is ZODB5-only interface. Bug introduced in
dd959b28 (zodbdump += DumpReader - to read/parse zodbdump stream).

Currently we exercise zodbdump and zodbcommit+zodbdump with non-empty
extensions, which works if ZODB is patched for txn.extension_bytes
support, but fails on pristine ZODB.

Support for txn.extension_bytes cannot get into upstream ZODB for
more than a year:

https://github.com/zopefoundation/ZODB/pull/183
https://github.com/zopefoundation/ZODB/pull/207

and  even if it somehow will make it, it will likely be only in ZODB5,
while we still care to support ZODB4 and ZODB3.

Skipping zodbdump / zodbcommit tests, if a ZODB does not have
txn.extension_bytes support, would result in significant reduction of
zodbtools test coverage, because practically that is the current
situation with all upstream ZODB{3,4,5}. Dropping test coverage for
non-empty extensions is neither a good option.

For those reason, let's rework the tests and test both zodbdump and
zodbcommit with two scenarios:

1. on a test database where transactions extensions are always empty.
   This should work on all ZODB irregardless of whether
   txn.extension_bytes patch is there or not.

2. on a test database where transactions extensions are present.
   This should work if ZODB has txn.extension_bytes support, but if not,
   we can mark this case as xfail, since the failure is expected.

This way we make the testsuite pass irregardless of whether
txn.extension_bytes support is there, and we don't abandon dump/commit
testing coverage.

/helped-by Jérome Perrin <jerome@nexedi.com>

Fix for this kind of errors:

```
(env)$ zodb analyze demo.fs ffffffffffffffff..
# ø
Processed 0 records in 0 transactions
Traceback (most recent call last):
  File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/env/bin/zodb", line 11, in <module>
    load_entry_point('zodbtools', 'console_scripts', 'zodb')()
  File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodb.py", line 130, in main
    return command_module.main(argv)
  File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 305, in main
    report(analyze(path, use_dbm, delta_fs, tidmin, tidmax), csv)
  File "/srv/slapgrid/slappart8/srv/runner/project/zodbtools/zodbtools/zodbanalyze.py", line 102, in report
    print "Average record size is %7.2f bytes" % (rep.DBYTES * 1.0 / rep.OIDS)
ZeroDivisionError: float division by zero
```

and also small fixes for python3 compatibility

/reviewed-on nexedi/zodbtools!9

To keep a consistent output.

and use six.moves for python3 compatibility.

Previously we were using "anydbm" which selects dbhash, gdbm or dbm, but
opening the db with the f flag that's only valid for gdm, so de-facto we
were supporting only gdbm.

this also solves the following error on python3:

 AttributeError: 'dict_keys' object has no attribute 'sort'

/reviewed-on nexedi/zodbtools!10

- add help tidrange topic.
- change all commands to refer to it.
- add TODO to parse tid from absolute and relative dates (e.g.
  1.month.ago, similarly to how git can do). Dateparser

	https://dateparser.readthedocs.io/

  will probably be of help here.

/reviewed-on nexedi/zodbtools!7

Currently zodbanalyze analyzes whole storage. However it becomes
non practical to make a full zodbanalyze run on whole storage because
usually there are many transactions and objects and the time to run full
zodbanalyze is huge.

However, similarly to zodbdump, we can teach zodbanalyze to analyze a
particular range of transactions. This should help to analyze a range of
changes for e.g.  yesterday, or for last week or similar.

/reviewed-on nexedi/zodbtools!7

Analyze uses regular ZODB storage API: .iterator() & friends. This way
it should be possible apply it not only to FileStorage, but to other
type of storages as well - for example to NEO and ZEO.

Use zodbtools.util.storageFromURL to open a storage by knowing its URL.
Preserve support to directly apply zodbanalyze to FileStorage deltas.

/reviewed-on !7

Storages need to be closed to indicate a clean access end. If a storage is
not closed cleanly it might require to spend time and resources on next
open. For example FileStorage might need to recompute the index.

/reviewed-on nexedi/zodbtools!6