- 16 Apr, 2019 1 commit
-
-
Kirill Smelkov authored
As of Go1.12 the check for whether particular trace event is enabled or not is inlined into caller. For example: ---- 8< ---- (connection.go) //trace:event traceMsgSendPre(l *NodeLink, connId uint32, msg proto.Msg) func (link *NodeLink) sendMsg(connId uint32, msg proto.Msg) error { traceMsgSendPre(link, connId, msg) buf := msgPack(connId, msg) return link.sendPkt(buf) } ---- 8< ---- (ztrace.go) // traceevent: traceMsgSendPre(l *NodeLink, connId uint32, msg proto.Msg) type _t_traceMsgSendPre struct { tracing.Probe probefunc func(l *NodeLink, connId uint32, msg proto.Msg) } var _traceMsgSendPre *_t_traceMsgSendPre func traceMsgSendPre(l *NodeLink, connId uint32, msg proto.Msg) { if _traceMsgSendPre != nil { _traceMsgSendPre_run(l, connId, msg) } } func _traceMsgSendPre_run(l *NodeLink, connId uint32, msg proto.Msg) { for p := _traceMsgSendPre; p != nil; p = (*_t_traceMsgSendPre)(unsafe.Pointer(p.Next())) { p.probefunc(l, connId, msg) } } ---- 8< ---- (connection.o) TEXT lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·(*NodeLink).sendMsg(SB), ABIInternal, $40-48 // connection.go:1368 // MOVQ (TLS), CX (stack growth prologue) // CMPQ SP, 16(CX) // JLS 177 // SUBQ $40, SP // MOVQ BP, 32(SP) (BP save) // LEAQ 32(SP), BP (BP init) // FUNCDATA $0, gclocals·dbae954985b577993f0eafab1347dd21(SB) (args) FUNCDATA $1, gclocals·f6bd6b3389b872033d462029172c8612(SB) (locals) FUNCDATA $3, gclocals·47f0f3578d0f16ef296c57af2564832c(SB) PCDATA $2, $0 // connection.go:1369 PCDATA $0, $0 XCHGL AX, AX CMPQ lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·_traceMsgSendPre(SB), $0 <-- inlined traceMsgSendPre JNE pc130 // ztrace.go:50 <-- pc44: MOVL connId+56(SP), AX // connection.go:1371 <-- normal path MOVL AX, (SP) MOVQ msg+64(SP), AX MOVQ AX, 8(SP) PCDATA $2, $1 PCDATA $0, $1 MOVQ msg+72(SP), AX PCDATA $2, $0 MOVQ AX, 16(SP) CALL lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·msgPack(SB) PCDATA $2, $1 MOVQ 24(SP), AX PCDATA $2, $2 // connection.go:1372 PCDATA $0, $2 MOVQ link+48(SP), CX PCDATA $2, $1 MOVQ CX, (SP) PCDATA $2, $0 MOVQ AX, 8(SP) CALL lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·(*NodeLink).sendPkt(SB) PCDATA $2, $1 MOVQ 24(SP), AX MOVQ 16(SP), CX PCDATA $0, $3 MOVQ CX, _r2+80(SP) PCDATA $2, $0 MOVQ AX, _r2+88(SP) // MOVQ 32(SP), BP (BP restore) // ADDQ $40, SP (SP restore) RET pc130: <-- slow path (tracepoint attached) PCDATA $2, $1 // ztrace.go:51 PCDATA $0, $0 MOVQ link+48(SP), AX PCDATA $2, $0 MOVQ AX, (SP) MOVL connId+56(SP), CX MOVL CX, 8(SP) MOVQ msg+64(SP), DX MOVQ DX, 16(SP) PCDATA $2, $3 MOVQ msg+72(SP), BX PCDATA $2, $0 MOVQ BX, 24(SP) CALL lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·_traceMsgSendPre_run(SB) JMP pc44 // NOP (stack growth) // PCDATA $0, $-1 // connection.go:1368 // PCDATA $2, $-1 // CALL runtime.morestack_noctxt(SB) // JMP 0
-
- 12 Apr, 2019 1 commit
-
-
Kirill Smelkov authored
This continues b46436bd (tracing/runtime: Try to add support for Go1.12): Move `+build ...` constraint in assembly files to the top, as suggested by Russ Cox: https://github.com/golang/go/issues/31410#issuecomment-482599134 Then go stops complaining about it. The build constraint had no effect previously, but it was not caught because getg is used only on race builds and I was testing on amd64 only, which is practically almost the only ISA supported by race detector as of Go1.11 .
-
- 11 Apr, 2019 2 commits
-
-
Kirill Smelkov authored
Generate g for Go 1.12; no changes here compared to Go 1.11, however go1.12 build build fails because of +build in .s file: # lab.nexedi.com/kirr/go123/tracing/internal/xruntime ./runtime_g_amd64.s:3:1: +build comment must appear before package clause and be followed by a blank line Build problem reported upstream: https://github.com/golang/go/issues/31410
-
Kirill Smelkov authored
Pygolang, starting from 0.0.0.dev8, deprecates @method(cls) in favour of @func(cls): pygolang@942ee900
-
- 24 Jan, 2019 1 commit
-
-
Kirill Smelkov authored
- sqliteutil moved to -> sqlitex (https://github.com/crawshaw/sqlite/commit/e68ccf033b) - Poll moved from sqlite to sqlitex (https://github.com/crawshaw/sqlite/commit/1967107395) - Poll now takes full context, not only done channel (https://github.com/crawshaw/sqlite/commit/93b88b3c5c)
-
- 17 Jan, 2019 1 commit
-
-
Kirill Smelkov authored
-
- 16 Jan, 2019 1 commit
-
-
Kirill Smelkov authored
To -> pygolang@32a21d5b
-
- 19 Nov, 2018 1 commit
-
-
Kirill Smelkov authored
Pygolang, starting from 0.0.0.dev5 provides working defer.
-
- 06 Jul, 2018 1 commit
-
-
Kirill Smelkov authored
-
- 03 Jul, 2018 1 commit
-
-
Kirill Smelkov authored
We will hopefully need it in the future to test external python code similarly to when testing in-process go code with the help of tracepoints. See module description for details (tracetest is not yet done). There is no yet Go client API for interacting with such-traced python program. For the record: manually inspecting traced python process could be done via e.g. socat EXEC:"./pyruntraced 3 neo.trace.py -- ../../../t/backup-play/N1-writer",fdin=3,fdout=3 TCP:localhost:8888 and telnetting to localhost:8888 from another xterm.
-
- 20 Jun, 2018 1 commit
-
-
- 19 Jun, 2018 11 commits
-
-
Kirill Smelkov authored
This patch brings lonet implementation in Python with the idea that Go and Python programs could interoperate via lonet network and thus mixed Go/Python application cluster could be tested. Implementation quality is lower compared to Go version, but still it should be more or less ok.
-
Kirill Smelkov authored
Provide infrastructure, similar to what we already have in xerr, to Python programs: to wrap errors with error context and to extract the cause from a wrapped error. We will need it in the next patch.
-
Kirill Smelkov authored
Let's start with substitute for %q that is lacking in python. The implementation is copied from zodbtools from here: https://lab.nexedi.com/nexedi/zodbtools/blob/2801fae9/zodbtools/util.py#L39 It is not good for golang to depend on zodbtools and the function is minor. In the future maybe zodbtools in turn will change to import qq from golang.
-
Kirill Smelkov authored
- `go` spawns lightweight thread. - `chan` and `select` provide channels with Go semantic. - `method` allows to define methods separate from class. - `gimport` allows to import python modules by full path in a Go workspace. The focus of first draft was on usage interface and on correctness, not speed. In particular select should be fully working. If there is a chance I will maybe try to followup with gevent-based implementation in the future.
-
Kirill Smelkov authored
Lonet is the virtnet network that, contrary to pipenet, could be used when there are several OS-level processes involved. It uses SQLite for its registry and native OS-level TCP over loopback for data exchange. There is small text-based connection handshake protocol prelude that have to be carried out when a connection is tried to be established to host via its subnetwork, but after it data exchange goes directly through OS TCP stack. Lonet documentation follows: """ Package lonet provides TCP network simulated on top of localhost TCP loopback. For testing distributed systems it is sometimes handy to imitate network of several TCP hosts. It is also handy that ports allocated on Dial/Listen/Accept on that hosts be predictable - that would help tests to verify network events against expected sequence. When whole system could be imitated in 1 OS-level process, package lab.nexedi.com/kirr/go123/xnet/pipenet serves the task via providing TCP-like synchronous in-memory network of net.Pipes. However pipenet cannot be used for cases where tested system consists of 2 or more OS-level processes. This is where lonet comes into play: Similarly to pipenet addresses on lonet are host:port pairs and several hosts could be created with different names. A host is xnet.Networker and so can be worked with similarly to regular TCP network access-point with Dial/Listen/Accept. Host's ports allocation is predictable: ports of a host are contiguous integer sequence starting from 1 that are all initially free, and whenever autobind is requested the first free port of the host will be used. Internally lonet network maintains registry of hosts so that lonet addresses could be resolved to OS-level addresses, for example α:1 and β:1 to 127.0.0.1:4567 and 127.0.0.1:8765, and once lonet connection is established it becomes served by OS-level TCP connection over loopback. Example: net, err := lonet.Join(ctx, "mynet") hα, err := net.NewHost(ctx, "α") hβ, err := net.NewHost(ctx, "β") // starts listening on address "α:10" l, err := hα.Listen(":10") go func() { csrv, err := l.Accept() // csrv will have LocalAddr "α:1" }() ccli, err := hβ.Dial(ctx, "α:10") // ccli will be connection between "β:1" - "α:1" Once again lonet is similar to pipenet, but since it works via OS TCP stack it could be handy for testing networked application when there are several OS-level processes involved. """ """ Lonet organization For every lonet network there is a registry with information about hosts available on the network, and for each host its OS-level listening address. The registry is kept as SQLite database under /<tmp>/lonet/<network>/registry.db Whenever host α needs to establish connection to address on host β, it queries the registry for β and further talks to β on that address. Correspondingly when a host joins the network, it announces itself to the registry so that other hosts could see it. Handshake protocol After α establishes OS-level connection to β via main β address, it sends request to further establish lonet connection on top of that: > lonet "<network>" dial "<α:portα>" "<β:portβ>"\n β checks whether portβ is listening, and if yes, accepts the connection on corresponding on-β listener with giving feedback to α that connection was accepted: < lonet "<network>" connected "<β:portβ'>"\n After that connection is considered to be lonet-established and all further exchange on it is directly controlled by corresponding lonet-level Read/Write on α and β. If, on the other hand, lonet-level connection cannot be established, β replies: < lonet "<networkβ>" E "<error>"\n where <error> could be: - connection refused if <β:portβ> is not listening - network mismatch if β thinks it works on different lonet network than α - protocol error if β thinks that α send incorrect dial request - ... """
-
Kirill Smelkov authored
As we are going to implement another virtual network it would be good to share common code between implementations. For this generalize pipenet implementation to also cover the case when one does not own full network and owns only some hosts of it. An example of such situation is when one process handles one group of virtual hosts and another process handles another group of virtual hosts. Below a group of virtual hosts handled as part of network is called subnetwork. If hosts are not created in one place, we need a way to communicate information about new hosts in between subnetworks. This leads to using some kind of "registry" (see Registry interface). Then for the common code to be reused by a virtual network implementation it has to provide its working in the form of Engine interface to that common code. In turn the common code exposes another - Notifier - interface for particular network implementation to notify common code of events that come from outside to the subnetwork. Pipenet is reworked to be just a client of the common virtnet infrastructure. Virtnet documentation follows: """ Package virtnet provides infrastructure for TCP-like virtual networks. For testing distributed systems it is sometimes handy to imitate network of several TCP hosts. It is also handy that ports allocated on Dial/Listen/Accept on that hosts be predictable - that would help tests to verify network events against expected sequence. Package virtnet provides infrastructure for using and implementing such TCP-like virtual networks. Using virtnet networks Addresses on a virtnet network are host:port pairs represented by Addr. A network conceptually consists of several SubNetworks each being home for multiple Hosts. Host is xnet.Networker and so can be worked with similarly to regular TCP network access-point with Dial/Listen/Accept. Host's ports allocation is predictable: ports of a host are contiguous integer sequence starting from 1 that are all initially free, and whenever autobind is requested the first free port of the host will be used. Virtnet ensures that host names are unique throughout whole network. To work with a virtnet network, one uses corresponding package for particular virtnet network implementation. Such packages provide a way to join particular network and after joining give back SubNetwork to user. Starting from SubNetwork one can create Hosts and from those exchange data throughout whole network. Please see package lab.nexedi.com/kirr/go123/xnet/pipenet for particular well-known virtnet-based network. Implementing virtnet networks To implement a virtnet-based network one need to implement Engine and Registry. A virtnet network implementation should provide Engine and Registry instances to SubNetwork when creating it. The subnetwork will use provided engine and registry for its operations. A virtnet network implementation receives instance of Notifier together with SubNetwork when creating it. The network implementation should use provided Notifier to notify the subnetwork to handle incoming events. Please see Engine, Registry and Notifier documentation for details. """ Another virtnet-based network that is not limited to be used only in 1 OS process will follow next.
-
Kirill Smelkov authored
- clarify docstrings; - unexport NetPrefix.
-
Kirill Smelkov authored
On TCP port(accepted) = port(listen), i.e. When we connect www.nexedi.com:80, remote addr of socket will have port 80. Likewise on server side accepted socket will have local port 80. The connection should be thus fully identified by src-dst address pair. However currently pipenet, when accepting connection, allocates another free port at acceptor side.
-
Kirill Smelkov authored
Quoting documentation: """ Package xcontext provides addons to std package context. Merging contexts Merge could be handy in situations where spawned job needs to be canceled whenever any of 2 contexts becomes done. This frequently arises with service methods that accept context as argument, and the service itself, on another control line, could be instructed to become non-operational. For example: func (srv *Service) DoSomething(ctx context.Context) (err error) { defer xerr.Contextf(&err, "%s: do something", srv) // srv.serveCtx is context that becomes canceled when srv is // instructed to stop providing service. origCtx := ctx ctx, cancel := xcontext.Merge(ctx, srv.serveCtx) defer cancel() err = srv.doJob(ctx) if err != nil { if ctx.Err() != nil && origCtx.Err() == nil { // error due to service shutdown err = ErrServiceDown } return err } ... } """
-
Kirill Smelkov authored
- clarify Networker.Name on example of NetPlain; - we don't need Networker.Addr since we have Networker.Name; - format Dial/Listen docstrings so that they render properly under godoc; - format list in NetTLS doc so that it renders properly under godoc.
-
Kirill Smelkov authored
profile/trace outputs + files left after `go test -c`.
-
- 11 May, 2018 3 commits
-
-
Kirill Smelkov authored
As of go 47be3d49 there are 2 new types in g compared to go1.10: waitReason and ancestorInfo. Preliminary because Go1.11 has not been released yet, so zruntime_g_go1.11.go will need to be potentially regenerated after the release.
-
Kirill Smelkov authored
With previous sed expression it was failing e.g. on runtime.guintptr, giving much more after type definition: $ go doc -c -u runtime.muintptr | sed -n -e '/^type /,/^}/p' type muintptr uintptr muintptr is a *m that is not tracked by the garbage collector. Because we do free Ms, there are some additional constrains on muintptrs: 1. Never hold an muintptr locally across a safe point. 2. Any muintptr in the heap must be owned by the M itself so it can ensure it is not in use when the last true *m is released. func (mp muintptr) ptr() *m func (mp *muintptr) set(m *m) Fix it since we'll need to extract more single-line types for Go1.11 support. To verify it works de-stub {g,p,m}uintptr.
-
Kirill Smelkov authored
Go1.10 now modifies CGo sources as text, not AST, and this way comments are not removed and gotrace can see its '//trace:event ' comments in a CGo *.go files: https://github.com/golang/go/commit/85c3ebf4. This way with Go1.10 tests were failing because `//trace:event traceHello()` in a/pkg1/pkg1c.go was now noticed by gotrace and more then expected trace events produces. I have not noticed this in 65c399f0 (tracing/runtime: Add support for Go1.10 (preliminary)) probably because at that time above Go commit was not in my local Go1.10 tree. For the reference: tracing/runtime support regenerated for Go1.10.2 stays exactly the same as in 65c399f0 and so does not need updating.
-
- 10 May, 2018 2 commits
-
-
Kirill Smelkov authored
Commit 79e328c5 (xerr += First, Merge) was not right saying that if err == io.EOF will panic if err has dynamic type Errorv. It will not because interfaces with different dynamic types are always not equal. However Errorv == Errorv will indeed panic. Document and test that it is safe to compare error vectors to other errors, e.g. to io.EOF, and that Errorv == Errorv panics.
-
Kirill Smelkov authored
I was too used to my old Python habit while originally writing this. Using the occasion add package-level description which outlines the package to a user.
-
- 25 Apr, 2018 3 commits
-
-
Kirill Smelkov authored
We already have TraceConnect which corresponds to event of established network connection. However in many test scenarios it is required to know and pause execution right before dial is going to happen. For this introduce TraceDial which represents event corresponding to network dialing start.
-
Kirill Smelkov authored
Networker.Name should return name of access-point Networker represents on the network. We will need this information in the next patch for tracing dial events, when there is no connection established yet, and thus dialer location has to be taken from somewhere. It is also generally a good idea for Networker to have a name. For NetPlain networker the name is local hostname.
-
Kirill Smelkov authored
Should be in d3a7a196 (xnet/pipenet: Package for TCP-like synchronous in-memory network of net.Pipes)
-
- 24 Apr, 2018 1 commit
-
-
Kirill Smelkov authored
I was too used to my old Python habit while originally writing this. Using the occasion add package-level description which outlines the package to a user.
-
- 09 Apr, 2018 1 commit
-
-
Kirill Smelkov authored
Until https://github.com/golang/go/issues/7873 is fixed/implemented, let's follow the currently advised scheme to indent lists and this way put them as pre-formatted text. We already do so mostly everywhere, but 2 of the places were not following it and thus all items there were rendered as one line.
-
- 15 Mar, 2018 1 commit
-
-
Kirill Smelkov authored
-
- 15 Jan, 2018 1 commit
-
-
Kirill Smelkov authored
-
- 12 Jan, 2018 2 commits
-
-
Kirill Smelkov authored
For example in ZODB FileStorage format reader routines are written working with io.ReaderAt for the following reasons: - for loads random-access is required, - there can be several concurrent loads in flight simultaneously. At the same time various database iterations (APIs additional to load) use sequential access pattern and can be served by the same record reading routines. However with them we cannot use e.g. bufio.Reader because it works with plain io.Reader, not io.ReaderAt. Here comes SeqReaderAt: it adds a buffer, by default 2·4K, on top of original io.Reader, automatically detects direction of sequential access which can be forward, backward, or interleaved forward-backward patterns, and buffers data accordingly to avoid many syscalls e.g. in os.File case.
-
Kirill Smelkov authored
Currently by direct & buffered readers that also can report current offset in input stream. This functionality is handy to have when for example one needs to report an error after finding decoding problem and wants to include particular stream position in the report.
-
- 09 Jan, 2018 2 commits
-
-
Kirill Smelkov authored
Freelist of buffers is frequently needed to avoid GC/zeroing overhead in e.g. networked servers. Making buffer reference-counted is then required if buffer is shared between several users so that it is clear when it can go back to its pool. In ZODB this situation was arising on returning the same buffer from cached loads if several loads for same data are issued in close to parallel. [1] https://lab.nexedi.com/kirr/neo/tree/75a71514/go/zodb
-
Kirill Smelkov authored
-
- 21 Dec, 2017 2 commits
-
-
Kirill Smelkov authored
Preliminary because Go1.10 has not been released yet, so zruntime_g_go1.10.go will need to be potentially regenerated after the release.
-
Kirill Smelkov authored
./gotrace.go:707: Errorf format %s arg pkgi.Pkg.Path is a func value, not called
-