GitLab

Just a single change to expose time constants to C++ users (2476f47e).

This is top-level documentation for error chaining that was promised
and marked as TODO in

- fd95c88a (golang, errors, fmt: Error chaining (C++/Pyx))
- 17798442 (golang: Expose error at Py level)
- 78d0c76f (golang: Teach pyerror to be a base class)
- 337de0d7 (golang, errors, fmt: Error chaining (Python))
- 03f88c0b (errors: Take .__cause__ into account)

This provides top-level documentation for b and u that was promised and
marked as TODO in bcb95cd5 (golang: Provide b, u for strings).

The first step to expose errors and error chaining to Python:

- Add pyerror that wraps a pyx/nogil C-level error and is exposed as golang.error at py level.
- py errors must be compared by ==, not by "is"
- Add (py) errors.New to create a new error from text.
- a C-level error that has .Unwrap, is exposed with .Unwrap at py level,
  but full py-level chaining will be implemented in a follow-up patch.
- py error does not support inheritance yet.

Top-level documentation is TODO.

With Python3 I've got tired to constantly use .encode() and .decode();
getting exception if original argument was unicode on e.g. b.decode();
getting exception on raw bytes that are invalid UTF-8, not being able to
use bytes literal with non-ASCII characters, etc.

So instead of this pain provide two functions that make sure an object
is either bytes or unicode:

- b converts str/unicode/bytes s to UTF-8 encoded bytestring.

	Bytes input is preserved as-is:

	   b(bytes_input) == bytes_input

	Unicode input is UTF-8 encoded. The encoding always succeeds.
	b is reverse operation to u - the following invariant is always true:

	   b(u(bytes_input)) == bytes_input

- u converts str/unicode/bytes s to unicode string.

	Unicode input is preserved as-is:

	   u(unicode_input) == unicode_input

	Bytes input is UTF-8 decoded. The decoding always succeeds and input
	information is not lost: non-valid UTF-8 bytes are decoded into
	surrogate codes ranging from U+DC80 to U+DCFF.
	u is reverse operation to b - the following invariant is always true:

	   u(b(unicode_input)) == unicode_input

NOTE: encoding _and_ decoding *never* fail nor loose information. This
is achieved by using 'surrogateescape' error handler on Python3, and
providing manual fallback that behaves the same way on Python2.

The naming is chosen with the idea so that b(something) resembles
b"something", and u(something) resembles u"something".

This, even being only a part of strings solution discussed in [1],
should help handle byte- and unicode- strings in more robust and
distraction free way.

Top-level documentation is TODO.

[1] zodbtools!13

This release is driven by wendelin.core v2 needs with one of the changes
being that now most of the library was moved into nogil code and can be
used fully from inside nogil world(*). Python modules are now just wrappers
of their nogil counterparts. The way for Python and nogil worlds to
communicate is also provided.

The move to nogil required many other enhancements along the way. Please
see CHANGELOG for overview.

The move to nogil brought some speedup automatically.
Below are benchmark results of this release compared to pygolang v0.0.4
(1573d101) for python-level benchmarks (we have only those at present):

	(on i7@2.6GHz)

thread runtime:

    name             old time/op  new time/op  delta
    go               18.3µs ± 0%  18.3µs ± 1%     ~     (p=1.000 n=10+10)
    chan             2.91µs ± 3%  2.99µs ± 5%   +2.73%  (p=0.022 n=10+10)
    select           3.57µs ± 3%  3.57µs ± 4%     ~     (p=0.720 n=9+10)
    def              55.0ns ± 0%  54.0ns ± 0%   -1.82%  (p=0.002 n=8+10)
    func_def         43.8µs ± 2%  44.1µs ± 1%   +0.64%  (p=0.035 n=10+9)
    call             64.0ns ± 0%  66.3ns ± 1%   +3.59%  (p=0.000 n=10+10)
    func_call        1.05µs ± 1%  1.24µs ± 0%  +17.80%  (p=0.000 n=10+7)
    try_finally       138ns ± 0%   137ns ± 1%   -0.51%  (p=0.003 n=10+10)
    defer            2.32µs ± 1%  2.63µs ± 1%  +13.52%  (p=0.000 n=10+10)
    workgroup_empty  38.0µs ± 1%  24.1µs ± 1%  -36.43%  (p=0.000 n=10+10)
    workgroup_raise  47.7µs ± 1%  28.2µs ± 0%  -40.76%  (p=0.000 n=10+10)

gevent runtime:

    name             old time/op  new time/op  delta
    go               16.9µs ± 1%  17.2µs ± 2%   +1.94%  (p=0.000 n=10+10)
    chan             7.43µs ± 0%  7.82µs ± 0%   +5.34%  (p=0.000 n=10+7)
    select           10.5µs ± 0%  11.2µs ± 0%   +6.74%  (p=0.000 n=10+10)
    def              63.0ns ± 0%  57.6ns ± 1%   -8.57%  (p=0.000 n=9+10)
    func_def         44.0µs ± 1%  44.2µs ± 1%     ~     (p=0.063 n=10+10)
    call             67.0ns ± 0%  64.0ns ± 0%   -4.48%  (p=0.002 n=8+10)
    func_call        1.06µs ± 1%  1.23µs ± 1%  +16.50%  (p=0.000 n=10+10)
    try_finally       144ns ± 0%   136ns ± 0%   -5.90%  (p=0.000 n=10+10)
    defer            2.37µs ± 1%  2.61µs ± 1%  +10.07%  (p=0.000 n=10+10)
    workgroup_empty  57.0µs ± 0%  55.0µs ± 2%   -3.53%  (p=0.000 n=10+9)
    workgroup_raise  72.4µs ± 0%  69.6µs ± 6%   -3.95%  (p=0.035 n=9+10)

workgroup_* changes for thread runtime is the speedup I am talking about.
defer/func_call slowdown is due to added exception chaining. We did not
optimize Python-level defer yet, and if/when that would be needed, it should
be possible to optimize by moving pydefer implementation into Cython.

(*) go and channels were moved into nogil world in Pygolang v0.0.3 +
v0.0.4 . Now it is the rest of the library that was moved with packages
like context, time, sync etc.

wendelin.core v2 needs nogil to run pinner thread on client side to
support isolation property in cooperation with wcfs: since there is a
`client -> wcfs -> pinner` loop:

      - - - - - -
     |           |
        pinner <------.
     |           |   wcfs
        client -------^
     |           |
      - - - - - -
     client process

the pinner thread would deadlock if it tries to take the GIL because
client thread can be holding GIL already while accessing wcfs-mmaped
memory (think doing e.g. `x = A[i]` in Python).

A deferred function can raise exception and this exception itself can
have .__context__ - consider for example if B1 is raised and its chain
is B1->B2->B3. Before calling that deferred function, we save
then-current exception A1 into GoFrame.exc_ctx and link-to .exc_ctx after
the call. We were previously trying to link-to .exc_ctx from raised
exception itself - i.e. B1 in the above example which is not correct:
B1 was raised while B2 was being raised ... etc and it was B3 who was
raised after A1. The consequence was that A1 was list, since B1 already
had non-empty .__context__

-> Fix it by linking-to A1 from B3, not from B1. In other words by
linking-to .exc_ctx chain from tail of exception chain of raised
exception.

We can be sure that updated test is correct because it passes under
Python3 where exception chaining is implemented natively.

Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)

Defer support for exception chaining on Python2 (see bb9a94c3 "golang:
Teach defer to chain exceptions (PEP 3134) even on Python2") simulates
PEP 3134 by manually setting and chaining exc .__context__, .__cause__
and .__traceback__: if exceptions are thrown by several deferred
functions they are linked into exception chain via .__context__
attribute. Defer support in @func also makes sure that any exception
that comes out of a function has all those PEP 3134 attributes and
presets them to their default values if exception object did not have
them initially. In particular .__context__ is preset to None for first
raised exception.

There was a logic error in chaining handling: .__context__ was glued to
previously raised exception only if current exception object did not
have .__context__ attribute at all. And this was failing to chain them
if current exception was raised from under another function wrapped in
@func, because GoFrame wrapping that function makes sure to set
curexc.__context__=None and oops - it was not changed later.

-> Fix chaining implementation by gluing .__context__ either if there is
no such attribute in current exception, or if .__context__ is None.

This is correct to do since .__context__, by its definition, represents
implicitly / automatically chained exceptions - contrary to .__cause__
which is explicitly set and would be incorrect to automatically change
from None to something. We also know that the end result is the same as
Python3 behaviour since updated tests (see bellow) pass OK also when run
under Python3 where exceptions chaining is implemented by Python runtime
natively.

Update test_defer_excchain() in golang_test.py, which verifies how
raised exceptions are chained.

No need to update testprog/golang_test_defer_excchain.{py,txt} since the
test there verifies that our traceback dumper is correctly hooked into
Python interpreter - where we know that exceptions are already chained
correctly and we verify only that automatic traceback dump takes this
chaining into account.

Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)

Else the exception, even if it was recovered from, will be included as
cause for next raised exception. See added test for details.

Python3 chains exceptions, so that e.g. if exc1 is raised and, while it
was not handled, another exc2 is raised, exc2 will be linked to exc1 via
exc2.__context__ attribute and exc1 will be included into exc2 traceback
printout. However many projects still use Python2 and there is no
similar chaining functionality there. This way exc1 is completely lost.

Since defer code is in our hands, we can teach it to implement exception
chaining even on Python2 by carefully analyzing what happens in
_GoFrame.__exit__().

Implementing chaining itself is relatively easy, but is only part of the
story. Even if an exception is chained with its cause, but exception
dump does not show the cause, the chaining will be practically useless.
With this in mind this patches settles not only on implementing chaining
itself, but on also giving a promise that chained cause exceptions will
be included into traceback dumps as well.

To realize this promise we adjust all exception dumping funcitons in
traceback module and carefully install adjusted
traceback.print_exception() into sys.excepthook. This amends python
interactive sessions and programs run by python interpreter to include
causes in exception dumps. "Careful" here means that we don't change
sys.excepthook if on golang module load we see that sys.excepthook was already
changed by some other module - e.g. due to IPython session running
because IPython installs its own sys.excepthook. In such cases we don't
install our sys.excepthook, but we also provide integration patches that
add exception chaining support for traceback dump functionality in
popular third-party software. The patches (currently for IPython and
Pytest) are activated automatically, but only when/if corresponding
software is imported and actually used. This should give practically
good implementation of the promise - a user can now rely on seeing
exception cause in traceback dump whatever way python programs are run.

The implementation takes https://pypi.org/project/pep3134/ experience
into account [1]. peak.utils.imports [2,3] is used to be notified when/if
third-party module is imported.

[1] https://github.com/9seconds/pep3134/
[2] https://pypi.org/project/Importing/
[3] http://peak.telecommunity.com/DevCenter/Importing

This patch originally started as hacky workaround in wendelin.core
because in wcfs tests I was frequently hitting situations, where
exception raised by an assert was hidden by another exception raised in
further generic teardown check. For example wcfs tests check that wcfs
is unmounted after every test run [4] and if that fails it was hiding
problems raised by an assert. As the result I was constantly guessing
and adding code like [5] to find what was actually breaking. At some
point I added hacky workaround for defer to print cause exception not to
loose it [6]. [7] has more context and background discussion on this topic.

[4] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L70
[5] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L853-857
[6] kirr/wendelin.core@c00d94c7
[7] nexedi/zodbtools!13 (comment 81553)

After this patch, on Python2

    defer(cleanup1)
    defer(cleanup2)
    defer(cleanup3)
    ...

is no longer just a syntatic sugar for

    try:
        try:
            try:
                ...
            finally:
                cleanup3()
        finally:
            cleanup2()
    finally:
        cleanup1()

This release is bugfix-only. Compared to pygolang v0.0.3 (4ca65816)
the change in speed is likely within noise:

     (on i7@2.6GHz)

thread runtime:

    name             old time/op  new time/op  delta
    go               18.3µs ± 1%  18.3µs ± 1%    ~     (p=0.218 n=10+10)
    chan             2.97µs ± 5%  2.97µs ± 8%    ~     (p=0.781 n=10+10)
    select           3.59µs ± 2%  3.55µs ± 5%    ~     (p=0.447 n=9+10)
    def              56.0ns ± 0%  55.0ns ± 0%  -1.79%  (p=0.000 n=10+10)
    func_def         43.7µs ± 1%  43.8µs ± 1%  +0.35%  (p=0.029 n=10+10)
    call             65.0ns ± 0%  62.3ns ± 1%  -4.15%  (p=0.000 n=10+10)
    func_call        1.06µs ± 1%  1.04µs ± 0%  -1.26%  (p=0.000 n=10+8)
    try_finally       137ns ± 1%   137ns ± 0%    ~     (p=1.000 n=10+10)
    defer            2.32µs ± 0%  2.33µs ± 1%  +0.43%  (p=0.000 n=9+10)
    workgroup_empty  37.6µs ± 1%  37.1µs ± 2%  -1.29%  (p=0.003 n=10+10)
    workgroup_raise  47.9µs ± 1%  47.6µs ± 0%  -0.63%  (p=0.001 n=9+9)

gevent runtime:

    name             old time/op  new time/op  delta
    go               15.8µs ± 0%  16.1µs ± 1%  +2.18%  (p=0.000 n=9+10)
    chan             7.36µs ± 0%  7.21µs ± 0%  -1.97%  (p=0.000 n=8+10)
    select           10.4µs ± 0%  10.5µs ± 0%  +0.71%  (p=0.000 n=10+10)
    def              57.0ns ± 0%  55.0ns ± 0%  -3.51%  (p=0.000 n=10+10)
    func_def         43.3µs ± 1%  44.1µs ± 2%  +1.81%  (p=0.000 n=10+10)
    call             66.0ns ± 0%  65.0ns ± 0%  -1.52%  (p=0.000 n=10+10)
    func_call        1.04µs ± 1%  1.06µs ± 1%  +1.48%  (p=0.000 n=10+10)
    try_finally       137ns ± 1%   136ns ± 0%  -1.31%  (p=0.000 n=10+10)
    defer            2.32µs ± 0%  2.31µs ± 1%    ~     (p=0.472 n=8+10)
    workgroup_empty  56.0µs ± 0%  55.7µs ± 0%  -0.49%  (p=0.000 n=10+10)
    workgroup_raise  71.3µs ± 1%  71.7µs ± 1%  +0.62%  (p=0.001 n=10+10)

To denote that this function/classes work at Python level:

    - chan    -> pychan
    - select  -> pyselect
    - default -> pydefault
    - nilchan -> pynilchan

Plain code movement with just s/panic/pypanic/ as in golang.pyx panic is
aleady there and means semantically different thing. Moved code, even
though it lives in golang.pyx, is still Python code and requires Python
runtime and GIL. We'll be splitting channels implementation into nogil
world in the following patches.

Just plain movement to Cython brings the following speedup:

 (on i7@2.6GHz)

thread runtime:

    name             old time/op  new time/op  delta
    go               26.6µs ± 1%  21.7µs ± 1%  -18.54%  (p=0.000 n=10+10)
    chan             13.7µs ± 1%   9.9µs ± 4%  -27.80%  (p=0.000 n=10+10)
    select           29.3µs ± 2%  19.2µs ± 4%  -34.65%  (p=0.000 n=9+9)
    def              55.0ns ± 0%  58.0ns ± 0%   +5.45%  (p=0.000 n=10+10)
    func_def         44.0µs ± 1%  44.4µs ± 0%   +0.72%  (p=0.002 n=10+10)
    call             64.0ns ± 0%  63.0ns ± 0%   -1.56%  (p=0.002 n=8+10)
    func_call        1.09µs ± 1%  1.05µs ± 1%   -2.96%  (p=0.000 n=10+10)
    try_finally       139ns ± 2%   135ns ± 0%   -2.60%  (p=0.000 n=10+10)
    defer            2.36µs ± 1%  2.36µs ± 1%     ~     (p=0.617 n=10+10)
    workgroup_empty  58.1µs ± 1%  49.0µs ± 1%  -15.61%  (p=0.000 n=10+10)
    workgroup_raise  72.7µs ± 1%  62.6µs ± 1%  -13.88%  (p=0.000 n=10+10)

gevent runtime:

    name             old time/op  new time/op  delta
    go               28.6µs ± 0%  25.4µs ± 0%  -11.20%  (p=0.000 n=8+9)
    chan             15.8µs ± 1%  12.2µs ± 1%  -22.62%  (p=0.000 n=10+10)
    select           33.1µs ± 1%  23.3µs ± 2%  -29.60%  (p=0.000 n=10+10)
    def              55.0ns ± 0%  56.0ns ± 0%   +1.82%  (p=0.000 n=10+10)
    func_def         44.4µs ± 2%  43.0µs ± 1%   -3.06%  (p=0.000 n=10+9)
    call             64.0ns ± 2%  69.0ns ± 0%   +7.81%  (p=0.000 n=10+10)
    func_call        1.06µs ± 0%  1.06µs ± 1%     ~     (p=0.913 n=8+9)
    try_finally       136ns ± 0%   139ns ± 0%   +2.21%  (p=0.000 n=9+10)
    defer            2.29µs ± 1%  2.38µs ± 2%   +3.58%  (p=0.000 n=10+10)
    workgroup_empty  73.8µs ± 1%  70.5µs ± 1%   -4.48%  (p=0.000 n=10+10)
    workgroup_raise  94.1µs ± 0%  90.6µs ± 0%   -3.69%  (p=0.000 n=10+10)

- Add go functionality to libgolang runtime;
- Implement go for thread and gevent runtimes.

  * Thread runtime uses PyThread_start_new_thread which, if used
    carefully, does not depend on Python GIL and on e.g. POSIX reduces to
    tiny wrapper around pthread_create.

  * Gevent runtime uses gevent's Greenlet in Pyx mode. This turns
    gevent to be a build-time dependency.

- Provide low-level _taskgo in C client API;
- Provide type-safe C++-level go wrapper over _taskgo;
- Switch golang.go from py implementation into Pyx wrapper over
  Pyx/nogil API.

This is the first patch that adds Pyx/C++/C-level unit tests and hooks
them into golang_test.py .

This patch brings the following speedup to Python-level interface:

 (on i7@2.6GHz)

thread runtime:

    name             old time/op  new time/op  delta
    go               93.0µs ± 1%  26.6µs ± 1%  -71.41%  (p=0.000 n=10+10)
    chan             13.6µs ± 2%  13.7µs ± 1%     ~     (p=0.280 n=10+10)
    select           29.9µs ± 4%  29.3µs ± 2%   -1.89%  (p=0.017 n=10+9)
    def              61.0ns ± 0%  55.0ns ± 0%   -9.84%  (p=0.000 n=10+10)
    func_def         43.8µs ± 1%  44.0µs ± 1%   +0.66%  (p=0.006 n=10+10)
    call             62.5ns ± 1%  64.0ns ± 0%   +2.40%  (p=0.000 n=10+8)
    func_call        1.06µs ± 1%  1.09µs ± 1%   +2.72%  (p=0.000 n=10+10)
    try_finally       137ns ± 0%   139ns ± 2%   +1.17%  (p=0.033 n=10+10)
    defer            2.34µs ± 1%  2.36µs ± 1%   +0.84%  (p=0.015 n=10+10)
    workgroup_empty  96.1µs ± 1%  58.1µs ± 1%  -39.55%  (p=0.000 n=10+10)
    workgroup_raise   135µs ± 1%    73µs ± 1%  -45.97%  (p=0.000 n=10+10)

gevent runtime:

    name             old time/op  new time/op  delta
    go               68.8µs ± 1%  28.6µs ± 0%  -58.47%  (p=0.000 n=10+8)
    chan             14.8µs ± 1%  15.8µs ± 1%   +6.19%  (p=0.000 n=10+10)
    select           32.0µs ± 0%  33.1µs ± 1%   +3.25%  (p=0.000 n=10+10)
    def              58.0ns ± 0%  55.0ns ± 0%   -5.17%  (p=0.000 n=10+10)
    func_def         43.9µs ± 1%  44.4µs ± 2%   +1.20%  (p=0.007 n=10+10)
    call             63.5ns ± 1%  64.0ns ± 2%     ~     (p=0.307 n=10+10)
    func_call        1.08µs ± 1%  1.06µs ± 0%   -2.55%  (p=0.000 n=10+8)
    try_finally       142ns ± 0%   136ns ± 0%   -4.23%  (p=0.000 n=10+9)
    defer            2.32µs ± 1%  2.29µs ± 1%   -0.96%  (p=0.000 n=10+10)
    workgroup_empty  90.3µs ± 0%  73.8µs ± 1%  -18.29%  (p=0.000 n=10+10)
    workgroup_raise   108µs ± 1%    94µs ± 0%  -13.29%  (p=0.000 n=10+10)

(small changes are probably within noise; "go" and "workgroup_*" should be
representative)

Start transforming Pygolang from being pure Python project into project
that uses both Python and Cython and provides both Python and Cython API
to its users.

For Cython API there is a goal for it to be usable independently of
Python GIL - i.e. independently of Python runtime and limitations
imposed by Python's global interpreter lock. The motivation for this
work is wendelin.core v2, which, due to its design, would deadlock if it
tries to take the GIL in its pinner thread.

This commit brings bare minimum establishing libraries and build system.
It:

- Introduces C/C++ library libgolang. Libgolang will be serving most of
  nogil functionality and Pyx API will be a small wrapper over it. At
  this step Libgolang comes only with C-level panic;

- Introduces Pyx package golang that is aliased to Pyx package
  golang._golang . Users can do `from golang cimport ...` similarly to how
  they do py-level `from golang import ...`. At this step golang.pyx
  only wraps panic and provides way to transform C-level panic into Python
  exception;

- Introduces Py package golang.pyx.build . This package, similarly to
  golang.org/pkg/go/build, serves as the system to build Pyx packages -
  both external packages that use Pygolang in pyx mode, and to build
  Pygolang itself. The build, internally, is served by setuptools_dso
  and cython. An example how to use golang.pyx.build from external
  project comes in golang/pyx/testprog/golang_pyx_user/ and is used by
  tests to verify golang.pyx.build functionality;

- Introduces Pygolang build dependency to cython and setuptools_dso.

For buffered channel _tryrecv, on success, was unlocking ch._mu too
early - before accessing ch._dataq with ch._dataq.append().

Without the fix, newly added test breaks as e.g.

    golang/golang_test.py::test_chan_buf_recv_vs_tryrecv_race Exception in thread Thread-3:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 317, in _
        assert (_, _rx) == (1, None), ('i%d' % i)
    AssertionError: i30
    assert (0, None) == (1, None)
      At index 0 diff: 0 != 1
      Full diff:
      - (0, None)
      ?  ^
      + (1, None)
      ?  ^

For buffered channel _trysend, on success, was unlocking ch._mu too
early - before accessing ch._dataq with ch._dataq.popleft().

Without the fix, newly added test breaks as e.g.

    golang/golang_test.py::test_chan_buf_send_vs_tryrecv_race Exception in thread Thread-3:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 256, in _
        assert (_, _rx) == (1, None)
    AssertionError: assert (0, 209) == (1, None)
      At index 0 diff: 0 != 1
      Full diff:
      - (0, 209)
      + (1, None)

    Exception in thread Thread-2:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 754, in run
        self.__target(*self.__args, **self.__kwargs)
      File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 243, in _
        ch.send(i)
      File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 340, in send
        ok = self._trysend(obj)
      File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 417, in _trysend
        rx = self._dataq.popleft()
    IndexError: pop from an empty deque

We provide gpython since 32a21d5b (gpython: Python interpreter with
support for lightweight threads), and golang module, since the
beginning, automatically uses gevent if it was installed via monkey
patching.

With @func being a decorator, the following

	@func(cls)
	def name():
		...

is always processed by python as

	name = func(cls)(def name(): ...)

Before this patch it was leading to name being overridden with None:

	def f():
		print 'hello'

	class C:
		pass

	@func(C)
	def f(c):
		print 'C.f', c

	f()
	Traceback (most recent call last):
	  File "<console>", line 1, in <module>
	TypeError: 'NoneType' object is not callable

We can fix it by returning from `func(cls)(def name(): ...)` the
original `name` object from the calling context.

Unfortunately if `name` was not previously set I did not find a way(*) to
avoid polluting the calling namespace where it is set to what @func(cls)
returns (None) by the hardcoded way how python processes decorators:

	In [2]: c = """
	   ...: @fff
	   ...: def ccc():
	   ...:     return 1
	   ...: """

	In [3]: cc = compile(c, "file", "exec")

	In [4]: dis(cc)
	  2           0 LOAD_NAME                0 (fff)
	              3 LOAD_CONST               0 (<code object ccc at 0x7fafe58d0130, file "file", line 2>)
	              6 MAKE_FUNCTION            0
	              9 CALL_FUNCTION            1
	             12 STORE_NAME               1 (ccc)	<-- NOTE means: ccc = what fff() call returns
	             15 LOAD_CONST               1 (None)
	             18 RETURN_VALUE

At least with no overriding taking place the situation is better now.

NOTE: it is only @func(cls) which potentially pollutes calling
namespace. Just @func (without class) is always clean because by
definition it works as a regular decorator.

(*) there is a very low-level and potentially fragile way to disable
STORE_NAME after CALL_FUNCTION by dynamically patching caller's bytecode
at runtime and replacing STORE_NAME with POP_TOP + NOP...

- we are going to introduce golang.time, and from inside there without
  `from __future__ import absolute_imports` it won't be possible to import
  needed stdlib's time.

- we were already doing `from __future__ import print_function`, but
  only in some files.

-> It makes sense to apply updated __future__ usage uniformly.

- correct CHANGELOG title levels: since readme uses "----" for the
  first level and "~~~~" for the second level, "====" was interpreted as
  the third title level and "Pygolang change history" became
  sub-sub-section of "Additional packages and utilities" with section
  for each version conversely becoming first level. It was not very
  noticeable until 0c5f9d06 (readme: Push "Additional packages and
  utilities" into its own section) started to use "~~~~".
  -> Rework CHANGELOG titling to align with the one in README.

- fix minor markup bits in README.

Send/recv on the nil channel block forever; close panics.
If a nil channel is used in select - corresponding case is never selected.

Setting channel to nil is a usual idiom in Go to disable processing some
cases in select. Nil channel is also used as "done" for e.g.
context.Background() - for contexts that can be never canceled.

It was deprecated and scheduled to be removed in 942ee900 (golang:
Deprecate @method(cls) in favour of @func(cls)).

Since we already have func (see 5146eb0b "Add support for defer &
recover") we can use @func for both plain functions and for methods.

For example instead of

	@func
	def my_function(...):
	    ...

	@method(MyClass)			<-- NOTE
	def my_method(self, ...):
	    ...

have it as

	@func
	def my_function(...):
	    ...

	@func(MyClass)				<-- NOTE
	def my_method(self, ...):
	    ...

which looks more similar to Go and exposes less golang keywords to a user.

Provide gpython interpreter, that sets UTF-8 as default encoding,
integrates gevent and puts go, chan, select etc into builtin namespace.
Please see details in the documentation added by this patch.

pymain was taken from kirr/go123@7a476082
(go123: tracing/pyruntraced: New tool to run Python code with tracepoints activated (draft))

Move version definition from setup.py to golang module so that it is
available at runtime with plain golang.__version__ .

A comma was missing - this way last element was 'funcgimport', not
'func' and 'gimport', and so `from golang import *` was failing.

`defer` allows to schedule a cleanup to be executed when current function
returns. It is similar to `try`/`finally` but does not force the cleanup part
to be far away in the end. For example::

   wc = wcfs.join(zurl)    │     wc = wcfs.join(zurl)
   defer(wc.close)         │     try:
                           │        ...
   ...                     │        ...
   ...                     │        ...
   ...                     │     finally:
                           │        wc.close()

For completeness there is `recover` and `panic` that allow to program with
Go-style error handling, for example::

   def _():
      r = recover()
      if r is not None:
         print("recovered. error was: %s" % (r,))
   defer(_)

   ...

   panic("aaa")

But `recover` and `panic` are probably of less utility since they can be
practically natively modelled with `try`/`except`.

If `defer` is used, the function that uses it must be wrapped with `@func` or
`@method` decorators.

The implementation is partly inspired by work of Denis Kolodin:

- https://habr.com/post/191786
- https://stackoverflow.com/a/43028386/9456786

While the second phase of select is running we queue send/recv cases to
corresponding channels. At some point - when some of the cases are
already queued - a peer goroutine might try to send/recv on that
channel. And it will succeed because a waiter was queued to the channel.

At the same time select is continuing its enqueue loop and before enqueuing
to a channel it tries to send/recv there. If that channel became just ready
(i.e. just after select poll phase) the try to send/recv will succeed. This
means that actually 2 select cases could be executed at the same time.

Fix it by carefully checking whether some case already won before trying
to send/recv on a channel.

This fixes the test failures that were demonstrated by previous 2 patches.

There was no test for @method so far and that's why it went unnoticed.
But on Python3 it breaks on f.func_name:

	In [3]: def f(): pass

	In [4]: f.func_name
	---------------------------------------------------------------------------
	AttributeError                            Traceback (most recent call last)
	<ipython-input-4-662dcbac1531> in <module>()
	----> 1 f.func_name

	AttributeError: 'function' object has no attribute 'func_name'

Fix it by using f.__name__ which works on both py2 and py3.

Add test for @method to make sure it doesn't break unnoticed.

This is the Go behaviour, as demonstratd by the following program:

---- 8< ----
package main

import (
	"fmt"
	"time"
)

func work(w int) {
	for i := 0; ; i++ {
		fmt.Printf("w%d: %d\n", w, i)
		time.Sleep(1*time.Second)
	}
}

func main() {
	for i := 0; i < 100; i++ {
		go work(i)
	}

	time.Sleep(3*time.Second)
	println("main: exit")
}
---- 8< ----

Python3 complains that there is no .im_func & friends of a method.
Workaround it all with compat code.

Not only we can import modules by full path, but now we can also spawn
threads/coroutines and exchange data in between them with the same
primitives and semantic as in Go.

The bulk of new functionality is copied from here:

	kirr/go123@9e1aa6ab

Original commit description follows:

"""
golang: New _Python_ package to provide Go-like features to Python language
- `go` spawns lightweight thread.
- `chan` and `select` provide channels with Go semantic.
- `method` allows to define methods separate from class.
- `gimport` allows to import python modules by full path in a Go workspace.

The focus of first draft was on usage interface and on correctness, not speed.
In particular select should be fully working.

If there is a chance I will maybe try to followup with gevent-based
implementation in the future.
Hide whitespace changes
"""