* go's `when` does not support "chains" like `2.years.3.months.ago`.
* `10am noon` is ambigious, we don't want so support it.
* date only `2008-12-01` should not retain "current" hour:minute

Parsing dates is really slow.
On the other hand, this optimization might cause error if entering a
date looking like a TID, but I'm not sure there is.

- add help tidrange topic.
- change all commands to refer to it.
- add TODO to parse tid from absolute and relative dates (e.g.
  1.month.ago, similarly to how git can do). Dateparser

	https://dateparser.readthedocs.io/

  will probably be of help here.

/reviewed-on !7

Currently zodbanalyze analyzes whole storage. However it becomes
non practical to make a full zodbanalyze run on whole storage because
usually there are many transactions and objects and the time to run full
zodbanalyze is huge.

However, similarly to zodbdump, we can teach zodbanalyze to analyze a
particular range of transactions. This should help to analyze a range of
changes for e.g.  yesterday, or for last week or similar.

/reviewed-on nexedi/zodbtools!7

Analyze uses regular ZODB storage API: .iterator() & friends. This way
it should be possible apply it not only to FileStorage, but to other
type of storages as well - for example to NEO and ZEO.

Use zodbtools.util.storageFromURL to open a storage by knowing its URL.
Preserve support to directly apply zodbanalyze to FileStorage deltas.

/reviewed-on !7

Storages need to be closed to indicate a clean access end. If a storage is
not closed cleanly it might require to spend time and resources on next
open. For example FileStorage might need to recompute the index.

/reviewed-on nexedi/zodbtools!6

This should be in 7f0bbf7e (zodbdump: Add golden test).

Zodbcommit reads transaction description from stdin and commits read data into
ZODB. The transaction to be committed is read in zodbdump format, but without
first 'txn' header line. For example:

    user "author"
    description "change 123"
    extension ""
    obj 0000000000000001 4 null:00
    ZZZZ

This tool could be useful for testing and for low-level database
maintenance. Please see zodbcommit.py docstring for more details.

We will likely need this reader for `zodb restore` in the future.
We will also use this reader for `zodb commit` in the next patch.

pygolang dependency v↑ becuase we use recently introduced
golang.strconv to unquote user/desc/extension strings.

Python2 works. Python3 support is only minimal and incomplete.

hashRegistry will be needed for zodbdump reader to create a hasher by
its name. Since std hashlib does not have adler32/crc32 nor null - let's
add our own implementation for those hashers too.

The code is based on

	kirr/neo@a60c472c, and
	kirr/neo@3f578560

Before now we were emitting extra LF only in between transactions as a
separator. However the dump format states LF always goes after
transaction, and there is a reason for it:

	without LF in the end, it becomes ambiguous at EOF - whether it
	is a proper transaction end, or the transaction was cut.

So avoid the ambiguity by always emitting trailing LF after transaction
record.

I originally added escapeqq as part of 75c03368 (zodbdump: Start to
stabilize output format) with the task for this utility to quote string
into valid "..." string always quoted with ".

This utility was later copied to pygolang:

	kirr/pygolang@afa46cf5

and then further improved there to work under both Python2 and Python3
and to not escape printable UTF-8 characters:

	kirr/pygolang@02dddb97

So stop the duplication and simply switch to the better version.

We start to stabilize output format of `zodb dump`. It is actually now robust and the only thing I would contemplate to potentially change is to also cover transaction metadata by hash checksum. So please take a look at updated format (details in patch 1) to provide feedback because it is likely close to what it  will be in its final form.

We also add a program to generate test database which uses various fancy ZODB features and check `zodb dump` output on it to golden one (patch 3).

To be able to dump transaction metadata in raw form ZODB is patched a bit:

https://github.com/zopefoundation/ZODB/pull/183

and we try to detect whether appropriate support is there at runtime and if yes use it to streamline obtaining transaction extension as raw (patch 2).
Pleae see patch 1 (second half of `zodbdump.py` about what has to be taken on without such support and that it still can't work fully reliably).

/cc @nexedi
/reviewed-on nexedi/zodbtools!3

We add a program to generate a test database with all fancy features and
then check `zodb dump` output on it to golden on.

The test database itself is commited to git because we want to make sure
zodbdump works ok for particular exact data and because transaction
extension dict is potentially saved differently on various runs.

Quoting

  https://docs.python.org/2.7/library/stdtypes.html#dict.items    and
  https://docs.python.org/3.7/library/stdtypes.html#dictionary-view-objects

""" CPython implementation detail: Keys and values are listed in an arbitrary
    order which is non-random, varies across Python implementations, and depends
    on the dictionary’s history of insertions and deletions. """

This way on test/gen_testdata.py changes it has to be run manually, and
then the output result of the run committed back together with
gen_testdata.py changes.

This patch test at runtime whether used storage can provide transaction
metadata in raw form and if so used thi form directly without going
through long fragile way of stable pickling extension back to bytes.

Corresponding ZODB support is at

https://github.com/zopefoundation/ZODB/pull/183

Since zodbdump start (c0a6299f "zodbdump - Tool to dump content of a
ZODB database   (draft)") and up till now zodbdump output format was not
good. For example user and description transaction properties were
output without proper quoting, which in situation when there would be
fancy characters in there would break the output.

So start the format stabilization:

- user and description are output as quoted, so now they are guaranteed
  to be on one line. The quoting character is always " (instead of e.g.
  smartly quoting either by ' or " as python does) for easier
  compatibility with ZODB implementations in other languages.

- transaction extension is now printed as raw bytes, not as dict.
  The idea here is that `zodb dump`

  * should perform dump of raw data as stored inside ZODB so that later
    `zodb restore` could restore the database identically to the same state.

  * we should dump raw data instead of unpickled ones because generally
    on-disk extension's format can be any raw bytes and this information
    should be preserved.

- transaction status is now also output as quoted to preserve line
  breakage on fancy status codes.

- it is documented that sha1 is not the only allowed hash function that
  might be used.

- in hashonly mode we add trailing " -" to obj string so that it is
  possible to distinguish outputs of `zodb dump` and `zodb dump -hashonly`
  without knowing a-priory the way it was produced.

  The reason to do so is that it would be not good to e.g. by accident
  feed hashonly output to (future) `zodb restore`, which, without having
  a way to see it should not consume object data would read following
  transaction information as raw object data with confusing later
  errors (and a small chance to restore completely different database
  without reporting error at all).

Because ZODB iteration API gives us already unpickled extension and only
that, for now to dump it as raw we get a long road to pickle it back
also caring to try to pickle in stable order.

Hopefully this will be only a fallback because of

https://github.com/zopefoundation/ZODB/pull/183

and next zodbtools patch.

~~~~

For testing purposes (currently only quoting function unit test) py.test
usage is introduced.

The code is also generally polished here and there.