golang: Teach defer to chain exceptions (PEP 3134) even on Python2

Python3 chains exceptions, so that e.g. if exc1 is raised and, while it
was not handled, another exc2 is raised, exc2 will be linked to exc1 via
exc2.__context__ attribute and exc1 will be included into exc2 traceback
printout. However many projects still use Python2 and there is no
similar chaining functionality there. This way exc1 is completely lost.

Since defer code is in our hands, we can teach it to implement exception
chaining even on Python2 by carefully analyzing what happens in
_GoFrame.__exit__().

Implementing chaining itself is relatively easy, but is only part of the
story. Even if an exception is chained with its cause, but exception
dump does not show the cause, the chaining will be practically useless.
With this in mind this patches settles not only on implementing chaining
itself, but on also giving a promise that chained cause exceptions will
be included into traceback dumps as well.

To realize this promise we adjust all exception dumping funcitons in
traceback module and carefully install adjusted
traceback.print_exception() into sys.excepthook. This amends python
interactive sessions and programs run by python interpreter to include
causes in exception dumps. "Careful" here means that we don't change
sys.excepthook if on golang module load we see that sys.excepthook was already
changed by some other module - e.g. due to IPython session running
because IPython installs its own sys.excepthook. In such cases we don't
install our sys.excepthook, but we also provide integration patches that
add exception chaining support for traceback dump functionality in
popular third-party software. The patches (currently for IPython and
Pytest) are activated automatically, but only when/if corresponding
software is imported and actually used. This should give practically
good implementation of the promise - a user can now rely on seeing
exception cause in traceback dump whatever way python programs are run.

The implementation takes https://pypi.org/project/pep3134/ experience
into account [1]. peak.utils.imports [2,3] is used to be notified when/if
third-party module is imported.

[1] https://github.com/9seconds/pep3134/
[2] https://pypi.org/project/Importing/
[3] http://peak.telecommunity.com/DevCenter/Importing

This patch originally started as hacky workaround in wendelin.core
because in wcfs tests I was frequently hitting situations, where
exception raised by an assert was hidden by another exception raised in
further generic teardown check. For example wcfs tests check that wcfs
is unmounted after every test run [4] and if that fails it was hiding
problems raised by an assert. As the result I was constantly guessing
and adding code like [5] to find what was actually breaking. At some
point I added hacky workaround for defer to print cause exception not to
loose it [6]. [7] has more context and background discussion on this topic.

[4] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L70
[5] https://lab.nexedi.com/kirr/wendelin.core/blob/49e73a6d/wcfs/wcfs_test.py#L853-857
[6] kirr/wendelin.core@c00d94c7
[7] nexedi/zodbtools!13 (comment 81553)

After this patch, on Python2

    defer(cleanup1)
    defer(cleanup2)
    defer(cleanup3)
    ...

is no longer just a syntatic sugar for

    try:
        try:
            try:
                ...
            finally:
                cleanup3()
        finally:
            cleanup2()
    finally:
        cleanup1()

MANIFEST.in

 include COPYING README.rst CHANGELOG.rst tox.ini pyproject.toml trun
 include golang/libgolang.h
 include golang/runtime/libgolang.cpp
-recursive-include golang    *.py *.pxd *.pyx *.toml
+recursive-include golang    *.py *.pxd *.pyx *.toml *.txt
 recursive-include gpython   *.py
 recursive-include 3rdparty  *.h

README.rst

@@ -142,6 +142,12 @@ to be far away in the end. For example::
    ...                     │     finally:
                            │        wc.close()
 
+If deferred cleanup fails, previously unhandled exception, if any, won't be
+lost - it will be chained with (`PEP 3134`__) and included into traceback dump
+even on Python2.
+
+__ https://www.python.org/dev/peps/pep-3134/
+
 For completeness there is `recover` and `panic` that allow to program with
 Go-style error handling, for example::
 

golang/__init__.py

@@ -37,7 +37,7 @@ __all__ = ['go', 'chan', 'select', 'default', 'nilchan', 'defer', 'panic', 'reco
 
 from golang._gopath import gimport  # make gimport available from golang
 import inspect, sys
-import decorator
+import decorator, six
 
 
 # @func is a necessary decorator for functions for selected golang features to work.
@@ -113,6 +113,12 @@ class _GoFrame:
         self.deferv    = []     # defer registers funcs here
         self.recovered = False  # whether exception, if there was any, was recovered
 
+        # py2: to-be next exception in exception chain (PEP 3134)
+        if six.PY2:
+            self.exc_ctx    = None  # exception context to chain new exception into
+            self.exc_ctx_tb = None  # exc_tb we got when catching .exc_ctx.
+                                    # we will set .exc_ctx.__traceback__ to this
+                                    # if/when .exc_ctx will be chained into.
     def __enter__(self):
         pass
 
@@ -121,6 +127,26 @@ class _GoFrame:
         if exc_val is not None:
             __goframe__.recovered = False
 
+        # py2: simulate exception chaining (PEP 3134)
+        if six.PY2:
+            if exc_val is not None:
+                if not hasattr(exc_val, '__context__'):
+                    exc_val.__context__   = __goframe__.exc_ctx
+                if not hasattr(exc_val, '__cause__'):
+                    exc_val.__cause__     = None
+                if not hasattr(exc_val, '__suppress_context__'):
+                    exc_val.__suppress_context__ = False
+
+                # set .__traceback__ only for chained-to exceptions. top-level
+                # raised exception must remain without __traceback__, because
+                # if it was not yet caught, setting __traceback__ here early
+                # will be wrong compared to what sys.exc_info() returns in
+                # caller except block.
+                if __goframe__.exc_ctx is not None:
+                    __goframe__.exc_ctx.__traceback__ = __goframe__.exc_ctx_tb
+                __goframe__.exc_ctx    = exc_val
+                __goframe__.exc_ctx_tb = exc_tb
+
         if len(__goframe__.deferv) != 0:
             d = __goframe__.deferv.pop()
 
@@ -149,6 +175,11 @@ def recover():
     _, exc, _ = sys.exc_info()
     if exc is not None:
         goframe.recovered = True
+        # recovered: clear current exception context
+        if six.PY2:
+            goframe.exc_ctx    = None
+            goframe.exc_ctx_tb = None
+
     if type(exc) is _PanicError:
         exc = exc.args[0]
     return exc
@@ -168,6 +199,77 @@ def defer(f):
     goframe.deferv.append(f)
 
 
+# py2: defer simulates exception chaining. Adjust traceback.print_exception()
+# and default sys.excepthook so that, out of the box, dump of chained exceptions
+# is printed with all details automatically.
+if six.PY2:
+    import traceback
+    _tb_print_exception = traceback.print_exception
+    def _print_exception(etype, value, tb, limit=None, file=None):
+        if file is None:
+            file = sys.stderr
+        def emitf(msg):
+            print(msg, file=file)
+        def recursef(etype, value, tb):
+            _print_exception(etype, value, tb, limit, file)
+
+        _emit_exc_context(value, emitf, recursef)
+        _tb_print_exception(etype, value, tb, limit, file)
+
+    _tb_format_exception = traceback.format_exception
+    def _format_exception(etype, value, tb, limit=None):
+        l = []
+        def emitf(msg):
+            l.append(msg+"\n")
+        def recursef(etype, value, tb):
+            l.extend(_format_exception(etype, value, tb, limit))
+
+        _emit_exc_context(value, emitf, recursef)
+        l += _tb_format_exception(etype, value, tb, limit)
+        return l
+
+    # _emit_exc_context emits traceback for exc cause/context if any.
+    #
+    # emitf is used to emit raw text.
+    # recursef is used to spawn processing on cause exception object.
+    def _emit_exc_context(exc, emitf, recursef):
+        ecause   = getattr(exc, '__cause__', None)
+        econtext = getattr(exc, '__context__', None)
+        if ecause is not None:
+            recursef(type(ecause), ecause, getattr(ecause, '__traceback__', None))
+            emitf("\nThe above exception was the direct cause of the following exception:\n")
+
+        elif econtext is not None and not getattr(exc, '__suppress_context__', False):
+            recursef(type(econtext), econtext, getattr(econtext, '__traceback__', None))
+            emitf("\nDuring handling of the above exception, another exception occurred:\n")
+
+    # patch traceback functions: in python2.7 all exception-related functions
+    # in traceback module use either tb.print_exception() or tb.format_exception().
+    # This way if we patch those two and someone uses e.g. tb.print_exc(),
+    # it will print exception with cause/context included.
+    traceback.print_exception  = _print_exception
+    traceback.format_exception = _format_exception
+
+    # adjust default sys.excepthook. Do this only if sys.excepthook was not already overridden.
+    # Two cases are possible here:
+    #   1) golang is imported in regular interpreter, possibly late in the process;
+    #   2) golang is imported early as part of gpython startup.
+    # For "2" when we get here the "pristine" precondition will be true, and so
+    # we'll get to adjust sys.excepthook . For "1" if sys.excepthook is
+    # pristine - it is safe to adjust. If sys.excepthook is not pristine - it
+    # is not safe to adjust, because e.g. `import golang` was run from an
+    # interactive IPython session and IPython already installed its own
+    # sys.excepthook. We don't adjust sys.excepthook in such case, but we also
+    # provide integration patches that add exception chaining support for
+    # traceback dump functionality in popular third-party software.
+    if sys.excepthook is sys.__excepthook__:
+        sys.excepthook = traceback.print_exception
+
+    # install pytest/ipython integration patches.
+    # each patch is activated only when/if corresponding software is imported and actually used.
+    import golang._patch.pytest_py2
+    import golang._patch.ipython_py2
+
 
 # ---- go + channels ----
 

golang/_patch/__init__.py

+# -*- coding: utf-8 -*-
+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""Package _patch contains patch infrastructure and patches that Pygolang
+applies automatically."""
+
+from __future__ import print_function, absolute_import
+
+from peak.util import imports   # thanks PJE
+import sys
+
+# `@afterimport(modname) def f(mod)` arranges for f to be called after when, if
+# at all, module modname will be imported.
+#
+# modname must be top-level module.
+def afterimport(modname):
+    if '.' in modname:
+        raise AssertionError("BUG: modname has dot: %r" % (modname,))
+    def _(f):
+        def patchmod(mod):
+            #print('patching %s ...' % (modname,))
+            f()
+
+        # XXX on pypy < 7.3 lazy-loading fails: https://bitbucket.org/pypy/pypy/issues/3099
+        #     -> import & patch eagerly
+        if 'PyPy' in sys.version and sys.pypy_version_info < (7,3):
+            try:
+                mod = __import__(modname)
+            except ImportError:
+                return  # module not available - nothing to patch
+            patchmod(mod)
+            return
+
+        imports.whenImported(modname, patchmod)
+        return f
+    return _

golang/_patch/ipython_py2.py

+# -*- coding: utf-8 -*-
+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""ipython: py2: pygolang integration patches."""
+
+from __future__ import print_function, absolute_import
+
+from golang import _patch
+import inspect
+
+# PY3 adds support for chained exceptions created by defer.
+#
+# It returns False by default (we are running under py2) except when called
+# from IPython/core/ultratb.*.structured_traceback() for which it pretends to
+# be running py3 if raised exception has .__cause__ .
+class PY3:
+    @staticmethod
+    def __nonzero__():
+        fcall = inspect.currentframe().f_back
+        if fcall.f_code.co_name != "structured_traceback":
+            return False    # XXX also check class/module?
+        exc = fcall.f_locals.get('evalue', None)
+        if exc is None:
+            return False
+        if not hasattr(exc, '__cause__'):
+            return False
+        return True
+
+@_patch.afterimport('IPython')
+def _():
+    from IPython.utils import py3compat
+    py3compat.PY3 = PY3()

golang/_patch/pytest_py2.py

+# -*- coding: utf-8 -*-
+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""pytest: py2: pygolang integration patches."""
+
+from __future__ import print_function, absolute_import
+
+from golang import _patch
+import inspect
+
+# _PY2 adds support for chained exceptions created by defer.
+#
+# It returns True by default (we are running under py2) except when called from
+# _pytest/_code/code.FormattedExcinfo.repr_excinfo() for which it pretends to
+# be running py3 if raised exception has .__cause__ .
+class _PY2:
+    @staticmethod
+    def __nonzero__():
+        fcall = inspect.currentframe().f_back
+        if fcall.f_code.co_name != "repr_excinfo":
+            return True     # XXX also check class/module?
+        exci = fcall.f_locals.get('excinfo', None)
+        if exci is None:
+            return True
+        exc = getattr(exci, 'value', None)
+        if exc is None:
+            return True
+        if not hasattr(exc, '__cause__'):
+            return True
+        return False
+
+@_patch.afterimport('_pytest')
+def _():
+    from _pytest._code import code
+    code._PY2 = _PY2()

golang/golang_test.py

@@ -22,10 +22,12 @@ from __future__ import print_function, absolute_import
 
 from golang import go, chan, select, default, nilchan, _PanicError, func, panic, defer, recover
 from golang import sync
-from pytest import raises, mark
-from os.path import dirname
-import os, sys, inspect, importlib
+from pytest import raises, mark, fail
+from _pytest._code import Traceback
+from os.path import dirname, realpath
+import os, sys, inspect, importlib, traceback, doctest
 from subprocess import Popen, PIPE
+import six
 from six.moves import range as xrange
 import gc, weakref
 
@@ -1219,6 +1221,235 @@ def test_deferrecover():
     assert v == [7, 2, 1]
 
 
+# verify that defer correctly establishes exception chain (even on py2).
+def test_defer_excchain():
+    # just @func/raise embeds traceback and adds ø chain
+    @func
+    def _():
+        raise RuntimeError("err")
+    with raises(RuntimeError) as exci:
+        _()
+
+    e = exci.value
+    assert type(e) is RuntimeError
+    assert e.args == ("err",)
+    assert e.__cause__      is None
+    assert e.__context__    is None
+    if six.PY3: # .__traceback__ for top-level exception is not set on py2
+        assert e.__traceback__  is not None
+        tb = Traceback(e.__traceback__)
+        assert tb[-1].name == "_"
+
+    # exceptions in deferred calls are chained
+    def d1():
+        raise RuntimeError("d1: aaa")
+    def d2():
+        1/0
+    def d3():
+        raise RuntimeError("d3: bbb")
+
+    @func
+    def _():
+        defer(d3)
+        defer(d2)
+        defer(d1)
+        raise RuntimeError("err")
+
+    with raises(RuntimeError) as exci:
+        _()
+
+    e3 = exci.value
+    assert type(e3) is RuntimeError
+    assert e3.args == ("d3: bbb",)
+    assert e3.__cause__     is None
+    assert e3.__context__   is not None
+    if six.PY3: # .__traceback__ of top-level exception
+        assert e3.__traceback__ is not None
+        tb3 = Traceback(e3.__traceback__)
+        assert tb3[-1].name == "d3"
+
+    e2 = e3.__context__
+    assert type(e2) is ZeroDivisionError
+    #assert e2.args == ("division by zero",) # text is different in between py23
+    assert e2.__cause__     is None
+    assert e2.__context__   is not None
+    assert e2.__traceback__ is not None
+    tb2 = Traceback(e2.__traceback__)
+    assert tb2[-1].name == "d2"
+
+    e1 = e2.__context__
+    assert type(e1) is RuntimeError
+    assert e1.args == ("d1: aaa",)
+    assert e1.__cause__     is None
+    assert e1.__context__   is not None
+    assert e1.__traceback__ is not None
+    tb1 = Traceback(e1.__traceback__)
+    assert tb1[-1].name == "d1"
+
+    e = e1.__context__
+    assert type(e) is RuntimeError
+    assert e.args == ("err",)
+    assert e.__cause__      is None
+    assert e.__context__    is None
+    assert e.__traceback__  is not None
+    tb = Traceback(e.__traceback__)
+    assert tb[-1].name == "_"
+
+# verify that traceback.{print_exception,format_exception} work on chained
+# exception correctly.
+def test_defer_excchain_traceback():
+    # tbstr returns traceback that would be printed for exception e.
+    def tbstr(e):
+        fout_print = six.StringIO()
+        traceback.print_exception(type(e), e, e.__traceback__, file=fout_print)
+        lout_format = traceback.format_exception(type(e), e, e.__traceback__)
+        out_print  = fout_print.getvalue()
+        out_format = "".join(lout_format)
+        assert out_print == out_format
+        return out_print
+
+    # raise without @func/defer - must be printed correctly
+    # (we patch traceback.print_exception & co on py2)
+    def alpha():
+        def beta():
+            raise RuntimeError("gamma")
+        beta()
+
+    with raises(RuntimeError) as exci:
+        alpha()
+    e = exci.value
+    if not hasattr(e, '__traceback__'): # py2
+        e.__traceback__ = exci.tb
+
+    assertDoc("""\
+Traceback (most recent call last):
+  File "PYGOLANG/golang/golang_test.py", line ..., in test_defer_excchain_traceback
+    alpha()
+  File "PYGOLANG/golang/golang_test.py", line ..., in alpha
+    beta()
+  File "PYGOLANG/golang/golang_test.py", line ..., in beta
+    raise RuntimeError("gamma")
+RuntimeError: gamma
+""", tbstr(e))
+
+
+    # raise in @func/chained defer
+    @func
+    def caller():
+        def q1():
+            raise RuntimeError("aaa")
+        defer(q1)
+        def q2():
+            raise RuntimeError("bbb")
+        defer(q2)
+        raise RuntimeError("ccc")
+
+    with raises(RuntimeError) as exci:
+        caller()
+    e = exci.value
+    if not hasattr(e, '__traceback__'): # py2
+        e.__traceback__ = exci.tb
+
+    assertDoc("""\
+Traceback (most recent call last):
+  File "PYGOLANG/golang/__init__.py", line ..., in _
+    return f(*argv, **kw)
+  File "PYGOLANG/golang/golang_test.py", line ..., in caller
+    raise RuntimeError("ccc")
+RuntimeError: ccc
+
+During handling of the above exception, another exception occurred:
+
+Traceback (most recent call last):
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    d()
+  File "PYGOLANG/golang/golang_test.py", line ..., in q2
+    raise RuntimeError("bbb")
+RuntimeError: bbb
+
+During handling of the above exception, another exception occurred:
+
+Traceback (most recent call last):
+  File "PYGOLANG/golang/golang_test.py", line ..., in test_defer_excchain_traceback
+    caller()
+  ...
+  File "PYGOLANG/golang/__init__.py", line ..., in _
+    return f(*argv, **kw)
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    d()
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    d()
+  File "PYGOLANG/golang/golang_test.py", line ..., in q1
+    raise RuntimeError("aaa")
+RuntimeError: aaa
+""", tbstr(e))
+
+    e.__suppress_context__ = True
+    assertDoc("""\
+Traceback (most recent call last):
+  File "PYGOLANG/golang/golang_test.py", line ..., in test_defer_excchain_traceback
+    caller()
+  ...
+  File "PYGOLANG/golang/__init__.py", line ..., in _
+    return f(*argv, **kw)
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    d()
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    d()
+  File "PYGOLANG/golang/golang_test.py", line ..., in q1
+    raise RuntimeError("aaa")
+RuntimeError: aaa
+""", tbstr(e))
+
+    e.__cause__ = e.__context__
+    assertDoc("""\
+Traceback (most recent call last):
+  File "PYGOLANG/golang/__init__.py", line ..., in _
+    return f(*argv, **kw)
+  File "PYGOLANG/golang/golang_test.py", line ..., in caller
+    raise RuntimeError("ccc")
+RuntimeError: ccc
+
+During handling of the above exception, another exception occurred:
+
+Traceback (most recent call last):
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    d()
+  File "PYGOLANG/golang/golang_test.py", line ..., in q2
+    raise RuntimeError("bbb")
+RuntimeError: bbb
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "PYGOLANG/golang/golang_test.py", line ..., in test_defer_excchain_traceback
+    caller()
+  ...
+  File "PYGOLANG/golang/__init__.py", line ..., in _
+    return f(*argv, **kw)
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    d()
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    d()
+  File "PYGOLANG/golang/golang_test.py", line ..., in q1
+    raise RuntimeError("aaa")
+RuntimeError: aaa
+""", tbstr(e))
+
+
+# verify that dump of unhandled chained exception traceback works correctly (even on py2).
+def test_defer_excchain_dump():
+    # run golang_test_defer_excchain.py and verify its output via doctest.
+    dir_testprog = dirname(__file__) + "/testprog"      # pygolang/golang/testprog
+    with open(dir_testprog + "/golang_test_defer_excchain.txt", "r") as f:
+        tbok = f.read()
+    retcode, stdout, stderr = _pyrun(["golang_test_defer_excchain.py"],
+                                cwd=dir_testprog, stdout=PIPE, stderr=PIPE)
+    assert retcode != 0
+    assert stdout == b""
+    assertDoc(tbok, stderr)
+
+
 # defer overhead.
 def bench_try_finally(b):
     def fin(): pass
@@ -1313,3 +1544,31 @@ def test_panics():
     # panic with expected argument
     with panics(123):
         panic(123)
+
+# assertDoc asserts that want == got via doctest.
+#
+# in want:
+# - PYGOLANG means real pygolang prefix
+# - empty lines are changed to <BLANKLINE>
+def assertDoc(want, got):
+    if isinstance(want, bytes):
+        want = want.decode('utf-8')
+    if isinstance(got, bytes):
+        got  = got .decode('utf-8')
+
+    # normalize got to PYGOLANG
+    dir_pygolang = realpath(dirname(__file__) + "/..")  # pygolang
+    got = got.replace(dir_pygolang, "PYGOLANG")
+
+    # ^$ -> <BLANKLINE>
+    while "\n\n" in want:
+        want = want.replace("\n\n", "\n<BLANKLINE>\n")
+
+    X = doctest.OutputChecker()
+    if not X.check_output(want, got, doctest.ELLIPSIS):
+        # output_difference wants Example object with .want attr
+        class Ex: pass
+        _ = Ex()
+        _.want = want
+        fail("not equal:\n" + X.output_difference(_, got,
+                    doctest.ELLIPSIS | doctest.REPORT_UDIFF))

golang/testprog/golang_test_defer_excchain.py

+#!/usr/bin/env python
+# Copyright (C) 2019  Nexedi SA and Contributors.
+#                     Kirill Smelkov <kirr@nexedi.com>
+#
+# This program is free software: you can Use, Study, Modify and Redistribute
+# it under the terms of the GNU General Public License version 3, or (at your
+# option) any later version, as published by the Free Software Foundation.
+#
+# You can also Link and Combine this program with other software covered by
+# the terms of any of the Free Software licenses or any of the Open Source
+# Initiative approved licenses and Convey the resulting work. Corresponding
+# source of such a combination shall include the source code for all other
+# software used.
+#
+# This program is distributed WITHOUT ANY WARRANTY; without even the implied
+# warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# See COPYING file for full licensing terms.
+# See https://www.nexedi.com/licensing for rationale and options.
+"""This program is used to test traceback dump of chained exceptions.
+
+When run it fails and Python should print the dump.
+The dump is verified by test driver against golang_test_defer_excchain.txt .
+"""
+
+from __future__ import print_function, absolute_import
+
+from golang import defer, func
+
+def d1():
+    raise RuntimeError("d1: aaa")
+def d2():
+    1/0
+def d3():
+    raise RuntimeError("d3: bbb")
+
+@func
+def main():
+    defer(d3)
+    defer(d2)
+    defer(d1)
+    raise RuntimeError("err")
+
+if __name__ == "__main__":
+    main()

golang/testprog/golang_test_defer_excchain.txt

+Traceback (most recent call last):
+  File "PYGOLANG/golang/__init__.py", line ..., in _
+    return f(*argv, **kw)
+  File "golang_test_defer_excchain.py", line 42, in main
+    raise RuntimeError("err")
+RuntimeError: err
+
+During handling of the above exception, another exception occurred:
+
+Traceback (most recent call last):
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    ...
+  File "golang_test_defer_excchain.py", line 31, in d1
+    raise RuntimeError("d1: aaa")
+RuntimeError: d1: aaa
+
+During handling of the above exception, another exception occurred:
+
+Traceback (most recent call last):
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    ...
+  File "golang_test_defer_excchain.py", line 33, in d2
+    1/0
+ZeroDivisionError: ...
+
+During handling of the above exception, another exception occurred:
+
+Traceback (most recent call last):
+  ... "golang_test_defer_excchain.py", line 45, in <module>
+    main()
+  ...
+  File "PYGOLANG/golang/__init__.py", line ..., in _
+    return f(*argv, **kw)
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    ...
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    ...
+  File "PYGOLANG/golang/__init__.py", line ..., in __exit__
+    ...
+  File "golang_test_defer_excchain.py", line 35, in d3
+    raise RuntimeError("d3: bbb")
+RuntimeError: d3: bbb

setup.py

@@ -224,7 +224,7 @@ setup(
                   ],
     include_package_data = True,
 
-    install_requires = ['gevent', 'six', 'decorator'],
+    install_requires = ['gevent', 'six', 'decorator', 'Importing;python_version<="2.7"'],
     extras_require   = extras_require,
 
     entry_points= {'console_scripts': [