- 23 Aug, 2019 16 commits
-
-
Kirill Smelkov authored
This test passes, but the functionality was not tested before. In particular all tests were passing even with the following hand-edits: --- a/golang/__init__.py +++ b/golang/__init__.py @@ -627,7 +627,7 @@ def selected(): sel = g.which if isinstance(sel, _SendWaiting): if not sel.ok: - panic("send on closed channel") + panic("send on closed channel ZZZ") return sel.sel_n, None if isinstance(sel, _RecvWaiting): with added tests the bug is caught and reported: def test_select(): N = 1000 # times to do repeated select/chan or select/select interactions # sync: close vs select(send) ch = chan() def _(): waitBlocked(ch.send) ch.close() go(_) > with panics("send on closed channel"): select((ch.send, 0)) golang_test.py:353: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <golang.golang_test.panics instance at 0x7fc1da66e5f0>, exc_type = <class 'golang._PanicError'>, exc_val = _PanicError('send on closed channel ZZZ',) exc_tb = <traceback object at 0x7fc1dabc33b0> def __exit__(self, exc_type, exc_val, exc_tb): ok = self.raises.__exit__(exc_type, exc_val, exc_tb) if not ok: return ok # _PanicError raised - let's check panic argument > assert self.exc_info.value.args == (self.arg,) E AssertionError: assert ('send on clo...channel ZZZ',) == ('send on closed channel',) E At index 0 diff: 'send on closed channel ZZZ' != 'send on closed channel' E Use -v to get the full diff golang_test.py:1032: AssertionError
-
Kirill Smelkov authored
For buffered channel _tryrecv, on success, was unlocking ch._mu too early - before accessing ch._dataq with ch._dataq.append(). Without the fix, newly added test breaks as e.g. golang/golang_test.py::test_chan_buf_recv_vs_tryrecv_race Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 317, in _ assert (_, _rx) == (1, None), ('i%d' % i) AssertionError: i30 assert (0, None) == (1, None) At index 0 diff: 0 != 1 Full diff: - (0, None) ? ^ + (1, None) ? ^
-
Kirill Smelkov authored
For buffered channel _trysend, on success, was unlocking ch._mu too early - before accessing ch._dataq with ch._dataq.popleft(). Without the fix, newly added test breaks as e.g. golang/golang_test.py::test_chan_buf_send_vs_tryrecv_race Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 256, in _ assert (_, _rx) == (1, None) AssertionError: assert (0, 209) == (1, None) At index 0 diff: 0 != 1 Full diff: - (0, 209) + (1, None) Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/kirr/src/tools/go/pygolang-master/golang/golang_test.py", line 243, in _ ch.send(i) File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 340, in send ok = self._trysend(obj) File "/home/kirr/src/tools/go/pygolang-master/golang/__init__.py", line 417, in _trysend rx = self._dataq.popleft() IndexError: pop from an empty deque
-
Kirill Smelkov authored
Currently it behaves correctly, but this aspect will need to be explicitly cared about when channel implementation moves to C. Add test to make sure it won't regress.
-
Kirill Smelkov authored
Starting from b51b8d5d (select: Run tests more thoroughly) we are running select subtests in repeated mode with N=1000. However not all select subtests were run in repeated mode - for example e.g. "non-blocking try send: not ok" was being run only once. -> Rework all select subtests to be run in repeated mode to increase the probability of catching bugs.
-
Kirill Smelkov authored
Channels implementation will soon be moved to C and will become independent from Python runtime. This way while pygolang will still be providing panic, hooking in and raising arbitrary Python-level exception will become problematic. -> Rework "blocks forever" tests to rely on just panic to prepare for that.
-
Kirill Smelkov authored
Chan implementation is going to be moved into C, and this way direct access to chan internals won't be possible. C implementation will export dedicated functions for chan tests which will be used. Prepare for that and factor out retrieving len of chan recv/send queues into separate functions, which will be adapted at the time of C port.
-
Kirill Smelkov authored
Nil channel will soon be changed to be represented by underlying NULL C pointer and there won't be channel object for nil and correspondingly recv/send queues won't be there. -> Prepare for that and don't check for recvq/sendq on nil channel.
-
Kirill Smelkov authored
We provide gpython since 32a21d5b (gpython: Python interpreter with support for lightweight threads), and golang module, since the beginning, automatically uses gevent if it was installed via monkey patching.
-
Kirill Smelkov authored
This currently pass trivially (since time.now = stdtime.now), but will be useful once we get time.now implementation into our hands and move it into Cython.
-
Kirill Smelkov authored
After 9c260fde (time: New package that mirrors Go's time) we have golang.time.now and golang.time.sleep and it makes it a more self-dependent system if timing facility is used through golang.time instead of outside std time module. For now this is only a "cleanness" change, but will become important once we start adding pyx-level nogil API to pygolang - there it will be important to use golang.time.* for correctness.
-
Kirill Smelkov authored
Starting from e6bea2cf (sync: New package that mirrors Go's sync) there is sync.WaitGroup and it was using threading.Event to signal that waitgroup's counter dropped to zero. threading.Event is implemented in Python's stdlib via threading.Condition, which in turn uses threading.Lock and list of waiters to implement its functionality. Which in turn is very similar to what golang.chan does internally with semaphore and e.g. recv queue. I noticed this while debugging sync test deadlock (see previous patch) and suspecting bug in threading.Event for a moment. While it turned there is no bug in threading.Event, it is better to rely on the common functionality for similar tasks, and pygolang's channel perfectly matches here the need to signal an event. By using our own code instead of stdlib's threading we are likely becoming less bug-prone. This also brings a small speedup: (on i5@1.80GHz) name old time/op new time/op delta workgroup_empty 239µs ± 1% 220µs ± 1% -7.62% (p=0.000 n=10+9) workgroup_raise 275µs ± 2% 264µs ± 2% -3.85% (p=0.000 n=10+10)
-
Kirill Smelkov authored
sync.WorkGroup test was doing ctx.done() wait from under test mutex, something like def _(ctx, i): with mu: ... if i == 0: raise RuntimeError() # to cause ctx cancel ctx.done().recv() # i=1 -> wait till ctx is canceled but it can be a deadlock if T(i=1) runs first and enters ctx.done().recv() before T(i=0) is run - then T(i=0) will block forever waiting to lock mu. This failure was not seen so far, probably because the time to go a new thread/goroutine is relatively high. However one of upcoming patches, where go is made faster, revealed this problem and, without the fix, sync.WorkGroup test was regularly deadlocking. The problem was there from sync.WorkGroup beginning - from 9ee7ba91 (sync += WorkGroup). Fix the deadlock by waiting for ctx.done() outside of mu.
-
Kirill Smelkov authored
9c61f254 (pygopath: Initial draft) has a "TODO py3 support", probably because https://stackoverflow.com/a/67692 shows different codes for Python2 and Python3. However today we have tox coverage for all Python2.7, Python3.6 and Python3.7 and everything works including gimport tests. -> Remove the TODO.
-
Kirill Smelkov authored
Currently gimport tests depend on module-level setup_module / teardown_module, which require for gimport tests to be in separate file. Redo the test not to depend on global module-level state and setup/teardown. This allows e.g. to merge gimport tests into golang_test.py if/when we want/need to.
-
Kirill Smelkov authored
Should be in 2aad64bb (golang: Add support for nil channel).
-
- 20 Jul, 2019 1 commit
-
-
Kirill Smelkov authored
In tests in places where the code checks that something panics, verify not only that _PanicError is raised, but also what was the argument passed to panic. This makes sure that tested code panics in expected place, not just panics "somewhere". To keep signal/noise ratio high introduce `panics` which is similar to `pytest.raises` and asserts that wrapped code panics with expected argument.
-
- 08 Jul, 2019 1 commit
-
-
Kirill Smelkov authored
-> Use .[test] to refer to them. https://stackoverflow.com/a/41398850/9456786
-
- 26 Jun, 2019 4 commits
-
-
Kirill Smelkov authored
A problem was hit with pytest.fail with raises Failed exception not being propagated to .wait. As it turned out it was not propagated because pytest's Failed derives from BaseException, not Exception, and we were catching only Exception and its children. Rework the code to propagate all exception types from workers. Performance change is with noise (it is either a bit faster for one set of runs, or a bit slower for another set of runs).
-
Kirill Smelkov authored
It is true for any decorator, that it makes things faster if def+decorator is used globally instead of at runtime, but for @func it is especially true since @func, using decorator.decorate, has relatively high overhead if it is used not only once at program startup, but instead every time something is run. In particular moving @func out of WorkGroup.go() make things considerably faster: name old time/op new time/op delta workgroup_empty 195µs ± 0% 113µs ± 1% -41.91% (p=0.008 n=5+5) workgroup_raise 221µs ± 1% 137µs ± 1% -38.29% (p=0.008 n=5+5) See https://lab.nexedi.com/kirr/misc/raw/009c4fee/pygolang/prof_workgroup_empty.svg for bench_workgroup_empty profile, where it is seen that decorator.decorate was using ~ half of the whole WorkGroup.go() time.
-
Kirill Smelkov authored
Start adding benchmarks to pygolang. Measure how much sync chan send/recv take and how much select takes for synchronous channels. Also measure how much overhead @func adds at both def and call times, and the overhead of defer compared to try/finally. For std (non-gevent'ed) Python2.7 we are currently at: name time/op go 91.6µs ± 1% chan 13.7µs ± 3% select 30.1µs ± 4% def 55.0ns ± 0% func_def 43.6µs ± 0% call 63.0ns ± 0% func_call 1.06µs ± 0% try_finally 136ns ± 1% defer 2.33µs ± 0%
-
Kirill Smelkov authored
@method(cls) was deprecated and removed in favour of @func(cls) in 942ee900 (golang: Deprecate @method(cls) in favour of @func(cls)) and 262f8986 (golang: Kill @method). The test name was overlooked.
-
- 21 Jun, 2019 1 commit
-
-
Kirill Smelkov authored
With @func being a decorator, the following @func(cls) def name(): ... is always processed by python as name = func(cls)(def name(): ...) Before this patch it was leading to name being overridden with None: def f(): print 'hello' class C: pass @func(C) def f(c): print 'C.f', c f() Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: 'NoneType' object is not callable We can fix it by returning from `func(cls)(def name(): ...)` the original `name` object from the calling context. Unfortunately if `name` was not previously set I did not find a way(*) to avoid polluting the calling namespace where it is set to what @func(cls) returns (None) by the hardcoded way how python processes decorators: In [2]: c = """ ...: @fff ...: def ccc(): ...: return 1 ...: """ In [3]: cc = compile(c, "file", "exec") In [4]: dis(cc) 2 0 LOAD_NAME 0 (fff) 3 LOAD_CONST 0 (<code object ccc at 0x7fafe58d0130, file "file", line 2>) 6 MAKE_FUNCTION 0 9 CALL_FUNCTION 1 12 STORE_NAME 1 (ccc) <-- NOTE means: ccc = what fff() call returns 15 LOAD_CONST 1 (None) 18 RETURN_VALUE At least with no overriding taking place the situation is better now. NOTE: it is only @func(cls) which potentially pollutes calling namespace. Just @func (without class) is always clean because by definition it works as a regular decorator. (*) there is a very low-level and potentially fragile way to disable STORE_NAME after CALL_FUNCTION by dynamically patching caller's bytecode at runtime and replacing STORE_NAME with POP_TOP + NOP...
-
- 24 May, 2019 1 commit
-
-
Kirill Smelkov authored
This reverts commit 469f21a9. Even though Debian stopped shipping Python3.6 I'm now building it myself to have test coverage for two latest Python releases.
-
- 16 May, 2019 7 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Implement deadlines / timeouts using timers added recently in 9c260fde (time: New package that mirrors Go's time).
-
Kirill Smelkov authored
In the next patch we'll be adding deadlines support with another test. Move common infrastructure that will be used in all context tests to be test-module global.
-
Kirill Smelkov authored
We'll need to mark contexts that miss their deadline with deadlineExceeded error in a follow-up patch. In this patch: transform _BaseCtx._cancel to accept an error with which a context has to be canceled, but for now always use canceled in places where _cancel is called.
-
Kirill Smelkov authored
Same story as in Go, but we were lacking documentation notice about this.
-
Kirill Smelkov authored
Add time.Timer, time.Ticker and convenience functions time.tick, time.after and time.after_func. These will be used in context to support deadlines and timeouts. While at time topic, also provide sleep and now from golang.time, so that there is no need to import both golang.time and stdlib's time in a file. Provide time constants in the module as they are useful to have and mirror constants provided by Go's time. Note: timers implementation is very suboptimal for now.
-
Kirill Smelkov authored
Python adds dirname of run program to sys.path . This way when golang_test_goleaked.py runs it can import modules located under golang/ just by their name. Until now this was not noticed, but in the next patch we are going to add golang.time module and if test program is run with golang/ in sys.path just plain `import time` won't import time from stdlib and instead import time from golang/ . Such behaviour can be mitigated by doing `from __future__ import absolute_import` and we do that, including in golang_test_goleaked.py (see 81dfefa0 "*: __future__ += absolute_imports; Use unified __future__ everywhere"). However that does not prevent modules - even modules from stdlib - who are doing `import time` and not doing future absolute_import to import golang's time instead of stdlib. For example on PyPy2 threading imports time and then the test breaks: Traceback (most recent call last): File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/golang_test_goleaked.py", line 24, in <module> from golang import go, chan File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/__init__.py", line 38, in <module> import inspect, threading, collections, random, sys File "/usr/lib/pypy/lib-python/2.7/threading.py", line 15, in <module> from time import time as _time, sleep as _sleep File "/home/kirr/src/tools/go/pygolang/.tox/pypy-thread/site-packages/golang/time.py", line 30, in <module> from golang import go, chan, select, default, nilchan, panic ImportError: cannot import name 'go' -> Move the test program into a directory different from golang/ to avoid this trap.
-
- 14 May, 2019 1 commit
-
-
Kirill Smelkov authored
- we are going to introduce golang.time, and from inside there without `from __future__ import absolute_imports` it won't be possible to import needed stdlib's time. - we were already doing `from __future__ import print_function`, but only in some files. -> It makes sense to apply updated __future__ usage uniformly.
-
- 10 May, 2019 1 commit
-
-
Kirill Smelkov authored
- correct CHANGELOG title levels: since readme uses "----" for the first level and "~~~~" for the second level, "====" was interpreted as the third title level and "Pygolang change history" became sub-sub-section of "Additional packages and utilities" with section for each version conversely becoming first level. It was not very noticeable until 0c5f9d06 (readme: Push "Additional packages and utilities" into its own section) started to use "~~~~". -> Rework CHANGELOG titling to align with the one in README. - fix minor markup bits in README.
-
- 09 May, 2019 1 commit
-
-
Kirill Smelkov authored
-
- 03 May, 2019 3 commits
-
-
Kirill Smelkov authored
WorkGroup provides way to spawn goroutines that work on a common task and wait for their completion. It is modelled after https://godoc.org/golang.org/x/sync/errgroup but is not equal to it. See WorkGroup docstring for details.
-
Kirill Smelkov authored
Add sync.Once and sync.WaitGroup.
-
Kirill Smelkov authored
Add .Context, .background .with_cancel and .with_value. There is no support for deadline/timeout yet, since Go time package is not yet mirrored. Contrary to Go stdlib version, we also support context.merge that takes https://godoc.org/lab.nexedi.com/kirr/go123/xcontext#hdr-Merging_contexts for its inspiration. See https://blog.golang.org/context for overview of contexts.
-
- 02 May, 2019 2 commits
-
-
Kirill Smelkov authored
We are going to add more packages to mirror Go stdlib (e.g. next is context). Organize a dedicated section for what is additionally provided besides mirroring Go language and refer to that place only once from the top-level abstract.
-
Kirill Smelkov authored
Send/recv on the nil channel block forever; close panics. If a nil channel is used in select - corresponding case is never selected. Setting channel to nil is a usual idiom in Go to disable processing some cases in select. Nil channel is also used as "done" for e.g. context.Background() - for contexts that can be never canceled.
-
- 24 Mar, 2019 1 commit
-
-
Kirill Smelkov authored
It was deprecated and scheduled to be removed in 942ee900 (golang: Deprecate @method(cls) in favour of @func(cls)).
-