- 19 Nov, 2019 5 commits
-
-
Kirill Smelkov authored
sync.PyWorkGroup will need to wrap raw Context into PyContext for spawned workers.
-
Kirill Smelkov authored
- Add PyError class that wraps Python exception; - change PyFunc::operator() return from void -> to error, which might be either PyError or special error indicating that Python interpreter is stopped. - Provide way to reraise that Python exception. We will need to see which exception, if any, PyFunc call raised in sync.PyWorkGroup, where this python-level exception will need to be propagated to PyWorkGroup.wait() and reraised from there.
-
Kirill Smelkov authored
pygil_ensure() was taking pyexitedMu and then GIL, which can deadlock via AB BA scenario if pygil_ensure() is also called in another thread with GIL already held. -> Fix it by always taking GIL with pyexitedMu released. Without the fix the test added always hangs with thread backend, e.g. like this: golang/pyx/runtime_test.py::test_pyx_pyfunc_vs_gil_deadlock STUCK terminate called after throwing an instance of 'golang::PanicError' what(): STUCK Fatal Python error: Aborted The bug was introduced in b073f6df (time: Move/Port timers to C++/Pyx nogil).
-
Kirill Smelkov authored
Commit b073f6df (time: Move/Port timers to C++/Pyx nogil) added C++ level PyFunc as a way to call Python function from under nogil code. We will soon need this functionality for sync.WorkGroup too, so move it into shared place. We move into regular library - not a pyx module (e.g. golang.pyx) - because for pyx modules linking is done at import time by specially generated code, while we need the linker support to automatically resolve e.g. calls to PyFunc::operator() from inside std::function.
-
Kirill Smelkov authored
In the next patch we are going to move PyFunc from time.pyx into shared library. Before doing that let's provide DSO visibility macros from libgolang.h to offload other libraries from duplicating dance on how to mark symbols visibility properly depending on compiler/OS.
-
- 18 Nov, 2019 2 commits
-
-
Kirill Smelkov authored
Gevent runtime uses python-level calls internally which might interfere with current python state. For example if current python exception is set, and e.g. go or makesema runtime call is made, the following happens: golang/golang_test.py::test_pyx_runtime_vs_pyexc RuntimeError: abc The above exception was the direct cause of the following exception: SystemError: <class 'gevent.__semaphore.Semaphore'> returned a result with an error set Exception ignored in: 'golang.runtime._runtime_gevent._sema_alloc' SystemError: <class 'gevent.__semaphore.Semaphore'> returned a result with an error set terminate called after throwing an instance of 'golang::PanicError' what(): makesema: alloc failed Fatal Python error: Aborted -> Fix all functions in the runtimes that work at Python level to save/restore Python-level exception on entry/exit. This is mostly gevent runtime, but also a couple of non-posix fallbacks in thread runtime. The bug was there from day 1 of runtimes - from ce8152a2 (pyx api: Provide sleep), f971a2a8 (pyx api: Provide go) and 69db91bf (libgolang: Add internal semaphores).
-
Kirill Smelkov authored
In the next patch we will fix runtime functions that work at Python level to preserve user Python exception state. Before that, as a preparatory step, restructure runtimes code, so that all such Python-accessing functions are grouped together.
-
- 15 Nov, 2019 1 commit
-
-
Kirill Smelkov authored
setuptools_dso hardcodes to use its own build_ext in its setup. We amend that build_ext and inject our own version carefully while setup() is run - see 7ae8c4f3 (pyx.build: Allow to combine C and C++ sources in one extension). However, currently, if user uses cmdclass = {'build_ext': ...} in his own setup, this will be ignored and overwritten by golang.pyx.build.build_ext . One example where this breaks things is wendelin.core which hooks into build_ext to first configure/build CCAN before any extension: https://lab.nexedi.com/nexedi/wendelin.core/blob/b26ba558/setup.py#L147-153 -> Fix it by taking into account what user could put into cmdclass.
-
- 14 Nov, 2019 17 commits
-
-
Kirill Smelkov authored
If defer was used multiple times in one function it was failing to compile. For example added test is failing like this: In file included from golang/runtime/libgolang_test.cpp:22: golang/runtime/libgolang_test.cpp: In function ‘void __test_defer(int*)’: ./golang/libgolang.h:457:59: error: redeclaration of ‘golang::_deferred _defer___COUNTER__’ #define defer(f) golang::_deferred _defer_ ## __COUNTER__ (f) ^ golang/runtime/libgolang_test.cpp:473:5: note: in expansion of macro ‘defer’ defer([&]() { ^~~~~ ./golang/libgolang.h:457:36: note: ‘golang::_deferred _defer___COUNTER__’ previously declared here #define defer(f) golang::_deferred _defer_ ## __COUNTER__ (f) ^~~~~~~ golang/runtime/libgolang_test.cpp:470:5: note: in expansion of macro ‘defer’ defer([&]() { ^~~~~ It turns out the __COUNTER__ was not expanded and they say you have to use 3 level of macro calls. Oh well... https://stackoverflow.com/a/27611869/9456786
-
Kirill Smelkov authored
std::function is frequently too long to type while func is more native to Go.
-
Kirill Smelkov authored
For example wendelin.core wants to accept Python-level context, transform it into C-level context and pass to pyx/nogil code.
-
Kirill Smelkov authored
Provide context-related functionality that can be used directly from C++ and Pyx/nogil codes. Python-level classes and functions become small wrappers around pyx/nogil ones. Like with timers (b073f6df "time: Move/Port timers to C++/Pyx nogil") and interfaces (5a99b769 "libgolang: Start providing interfaces") memory for objects dynamically allocated on heap is managed automatically.
-
Kirill Smelkov authored
Start new package that provides C++ amendments to be used by libgolang and its users. Current functionality: dict and set that wrap std::unordered_map and std::unordered_set into ergonomic interface. The code originates from wendelin.core: https://lab.nexedi.com/kirr/wendelin.core/blob/5a045ed1/wcfs/internal/wcfs_misc.h#L214-257 Pyx/nogil only.
-
Kirill Smelkov authored
With just one function - errors.New() - to create new error with given text. This will be soon used in Pyx/nogil context package to create context.canceled and context.deadlineExceeded errors. Pyx/nogil only for now.
-
Kirill Smelkov authored
- interface is analog of interface{} in Go. - error is analog of error in Go. For now the interfaces are implemented via classic C++ scheme via inheritance and additional vtable field in pointed object. In the future we might want to consider reworking that to Go-style interfaces without predeclaring which interfaces a class implements. Top-level documentation is TODO. Interfaces will be soon needed to describe Context interface in C context package.
-
Kirill Smelkov authored
- refptr==refptr (e.g. to compare whether returned error object is something particular) - adoptref() and newref() - object (exposed as gobject not to be confused with builtin pyx "object" referring to python object) All this will be soon used in C version of context package.
-
Kirill Smelkov authored
Briefly describe provided functionality instead of only referring to https://golang.org/pkg/context and https://blog.golang.org/context.
-
Kirill Smelkov authored
Thinking a bit more after e82b4fab (libgolang: Objects refcounting (initial draft)) and working on interfaces as part of context pyx -> C move, I've came to conclusion that the better name for on-heap objects managed by libgolang is just "object" without "ref" prefix. -> Rename refobj -> object and amend its documentation correspondingly.
-
Kirill Smelkov authored
So that refptr<T> could be used as keys in e.g. dict (unordered_map) or set (unordered_set). set<refptr> will soon be used by C version of context package.
-
Kirill Smelkov authored
We already provide select for array and std::initializer_list. However when cases are constructed at runtime dynamically both array and initializer_list are not convenient to use. -> Provide select() overload that accepts vector<_selcase>. This functionality will soon be used in C version of context package.
-
Kirill Smelkov authored
- indicated that WorkGroup is not provided in Go version (amends 40f3b90c "sync: Provide package documentation"). - fix typo in sync.pxd subject (amends 34b7a1f4 "golang: Expose Sema and Mutex as public Python and Cython/nogil API")
-
Kirill Smelkov authored
Every time tests are run under Python3 the following warnings are printed: golang/golang_test.py::test_func .../golang/golang_test.py:990: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec() assert inspect.formatargspec(*inspect.getargspec(MyClass.zzz)) == '(self, v, x=2, **kkkkwww)' golang/golang_test.py::test_func .../golang/golang_test.py:990: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly assert inspect.formatargspec(*inspect.getargspec(MyClass.zzz)) == '(self, v, x=2, **kkkkwww)' However since we are not going to drop Python2 support soon, and there is no reffered "use-instead" functionality on Python2, let's simply silence the warning for now.
-
Kirill Smelkov authored
It will be some extra work to define general equality when moving context code into C. However that is unnecessary as usually a key is specific to one particular package, and value associated with that key is struct, or some other aggregate, that is holding all package-specific context values. Switching to compare keys by identity preserves that use-case while making transition to C easier.
-
Kirill Smelkov authored
We always store just one key in there. This will simplify transition to C.
-
Kirill Smelkov authored
This continues 2c8063f4 (*: Channels must be compared by ==, not by "is" even for nilchan) and changes comparision of .done() for different contexts from "is" to ==. The reason is that soon we are going to move contexts into C, and this way in context tree there potentially can be PyCtx -> CCtx -> PyCtx chains, and while all those contexts could have the same C-level channel, it would be difficult to propagate the knowledge that Python-level pychan wrapper should be also exactly the same. We already treat channel objects as pointers and require to compare them by ==, so there should be no loss here. However to keep performance of repeated select( ctx.done(), ... ) we continue to require that .done() for a particular python context object returns exactly the same channel object every time.
-
- 13 Nov, 2019 1 commit
-
-
Kirill Smelkov authored
Rearrange time C++ API so that e.g. time.Ticker is already refptr<time._Ticker>. This way time users don't need to write refptr all the time.
-
- 11 Nov, 2019 2 commits
-
-
Kirill Smelkov authored
https://bitbucket.org/pypy/pypy/commits/251b47698e8e https://bitbucket.org/pypy/pypy/issues/3096#comment-54707695 Which in turn fixes recover to clear current exception (see 9e6ff8bd "golang: Fix recover to clear current exception") for context.
-
Kirill Smelkov authored
e.g. Context -> PyContext, with_cancel -> pywith_cancel, etc. This will make the followup transition of context to pyx/nogil more clear.
-
- 07 Nov, 2019 10 commits
-
-
Kirill Smelkov authored
As of 20191107 imp is present in Python 3.8 source tree and is _used_ there - see e.g. https://github.com/python/cpython/blob/v3.8.0-66-g7c20888e713/Lib/pkgutil.py#L188-L192 And they say imp will be there until at least Python 4 - see e.g. https://github.com/python/cpython/commit/e4f41deccf94 This fixes test_defer_excchain_dump failure under python3-dbg, where E Failed: not equal: E Differences (unified diff with -expected +actual): E @@ -1,4 +1,6 @@ E +PYGOLANG/golang/_gopath.py:37: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses E + import imp was added into traceback printed on stderr. Fixes: bb9a94c3 (golang: Teach defer to chain exceptions (PEP 3134) even on Python2)
-
Kirill Smelkov authored
Provide time.Ticker, time.Timer and friends that can be used directly from C++ and Pyx/nogil codes. Python-level classes become small wrapper around pyx/nogil ones. This is the first patch that moves to Pyx/nogil classes that are dynamically allocated on heap. refptr<T> is used to automatically manage lifetime of such objects. At Pyx level exposed API is very similar to Python-one, while internally it uses refptr<T> and friends.
-
Kirill Smelkov authored
Refobj was just added in previous patch as the class that provides base functionality for reference-counted objects.
-
Kirill Smelkov authored
Since we are going to move more pygolang functionality into C++ (timers, context, ...), and since C++ does not have garbage collector, we will need to find a way to automatically manage memory in leak/error free way. Let's do this via refptr<T> smart pointer (inspired by WebKit's RefPtr<T>), which, similarly to chan<T> automatically manages pointed object's lifetime through reference counting. refptr<T> will be used in follow-up patches. Top-level documentation is TODO.
-
Kirill Smelkov authored
This will be needed in Python-wrappers for C-level code which exposes channels. For example in time package whose implementation will soon be moved to nogil world fully.
-
Kirill Smelkov authored
3121b290 (golang: Teach pychan to work with channels of C types, not only PyObjects) added ability to convert pychan to chan[T], but described that ability only briefly in README. Provide more details in that description before we add description and implementation for reflexive pychan <- chan[T] wrapping.
-
Kirill Smelkov authored
We already have some sync functionality implemented in C++ (e.g. sync.Once, sync.WaitGroup) and we are going to add more and also move timers implementation to C++. It is getting crowded for that functionality to still live in libgolang.{h,cpp} -> Split sync & time functionality into their own C++ packages (still built-into libgolang.so)
-
Kirill Smelkov authored
Briefly describe provided functionality instead of only referring to https://golang.org/pkg/time.
-
Kirill Smelkov authored
Briefly describe provided functionality instead of only referring to https://golang.org/pkg/sync.
-
Kirill Smelkov authored
Make built extensions depend on golang/{sync,time}.pxd and rebuild if those files change. A bit overkill to depend globally, but more correct, since an extension that is using pygolang can be using those packages. Before the patch - without those depends - since distutils does not implement proper dependency tracking, extensions - even those that actually depend on sync/time pxd, were not rebuilt. sync.pxd was missed to be added in 34b7a1f4 (golang: Expose Sema and Mutex as public Python and Cython/nogil API). time.pxd was missed to be added in ce8152a2 (pyx api: Provide sleep).
-
- 06 Nov, 2019 2 commits
-
-
Kirill Smelkov authored
Just cosmetics - easier to oversee the code.
-
Kirill Smelkov authored
Ticker: self -> pytx Timer: self -> pyt
-