Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
N
neo
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Stefane Fermigier
neo
Commits
a00572f6
Commit
a00572f6
authored
Jan 11, 2018
by
Kirill Smelkov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
.
parent
5d928bd1
Changes
13
Show whitespace changes
Inline
Side-by-side
Showing
13 changed files
with
52 additions
and
55 deletions
+52
-55
go/neo/client/client.go
go/neo/client/client.go
+3
-3
go/neo/connection.go
go/neo/connection.go
+5
-5
go/neo/server/cluster_test.go
go/neo/server/cluster_test.go
+1
-1
go/zodb/cache.go
go/zodb/cache.go
+13
-17
go/zodb/cache_test.go
go/zodb/cache_test.go
+11
-11
go/zodb/storage/fs1/filestorage.go
go/zodb/storage/fs1/filestorage.go
+4
-4
go/zodb/storage/fs1/filestorage_test.go
go/zodb/storage/fs1/filestorage_test.go
+3
-3
go/zodb/str.go
go/zodb/str.go
+3
-3
go/zodb/zodb.go
go/zodb/zodb.go
+6
-5
go/zodb/zodbtools/catobj.go
go/zodb/zodbtools/catobj.go
+0
-1
go/zodb/zodbtools/dump.go
go/zodb/zodbtools/dump.go
+1
-1
go/zodb/zodbtools/dump_test.go
go/zodb/zodbtools/dump_test.go
+1
-1
go/zodb/zodbtools/info.go
go/zodb/zodbtools/info.go
+1
-0
No files found.
go/neo/client/client.go
View file @
a00572f6
...
...
@@ -74,7 +74,7 @@ var _ zodb.IStorageDriver = (*Client)(nil)
// NewClient creates new client node.
//
// It will connect to master @masterAddr and identify with s
ep
cified cluster name.
// It will connect to master @masterAddr and identify with s
pe
cified cluster name.
func
NewClient
(
clusterName
,
masterAddr
string
,
net
xnet
.
Networker
)
*
Client
{
cli
:=
&
Client
{
node
:
neo
.
NewNodeApp
(
net
,
neo
.
CLIENT
,
clusterName
,
masterAddr
,
""
),
...
...
@@ -479,7 +479,7 @@ func (c *Client) _Load(ctx context.Context, xid zodb.Xid) (*mem.Buf, zodb.Tid, e
return
buf
,
resp
.
Serial
,
nil
}
func
(
c
*
Client
)
Iterate
(
tidMin
,
tidMax
zodb
.
Tid
)
zodb
.
ITxnIterator
{
func
(
c
*
Client
)
Iterate
(
ctx
context
.
Context
,
tidMin
,
tidMax
zodb
.
Tid
)
zodb
.
ITxnIterator
{
// see notes in ../NOTES:"On iteration"
panic
(
"TODO"
)
}
...
...
@@ -511,7 +511,7 @@ func openClientByURL(ctx context.Context, u *url.URL, opt *zodb.OpenOptions) (zo
func
(
c
*
Client
)
URL
()
string
{
// XXX neos:// depending whether it was tls
// XXX options if such were given to open are discarded
// (but we need to be able to contruct URL if Client was created via NewClient directly)
// (but we need to be able to con
s
truct URL if Client was created via NewClient directly)
return
fmt
.
Sprintf
(
"neo://%s@%s"
,
c
.
node
.
ClusterName
,
c
.
node
.
MasterAddr
)
}
...
...
go/neo/connection.go
View file @
a00572f6
...
...
@@ -72,7 +72,7 @@ type NodeLink struct {
axdownFlag
atomic32
// 1 when AX is marked no longer operational
// axdown chan struct{} // ready when accept is marked as no longer operational
axdown1
sync
.
Once
// CloseAccept may be called several
l
times
axdown1
sync
.
Once
// CloseAccept may be called several times
down
chan
struct
{}
// ready when NodeLink is marked as no longer operational
downOnce
sync
.
Once
// shutdown may be due to both Close and IO error
...
...
@@ -478,7 +478,7 @@ func (c *Conn) downRX(errMsg *Error) {
for
{
// we set .rxdownFlag=1 above.
// now if serveRecv is outside `.rxq <- ...` critical section we know it is either:
// - before it -> it will eventually see .rxdownFlag=1 and won
t send pkt r
o rxq.
// - before it -> it will eventually see .rxdownFlag=1 and won
't send pkt t
o rxq.
// - after it -> it already sent pkt to rxq and won't touch
// rxq until next packet (where it will hit "before it").
//
...
...
@@ -509,7 +509,7 @@ func (c *Conn) downRX(errMsg *Error) {
// now if recvPkt is outside `... <- .rxq` critical section we know that it is either:
// - before it -> it will eventually see .rxdownFlag=1 and won't try to read rxq.
// - after it -> it already read pktfrom rxq and won't touch
// rxq until next recvPkt (where it will h
e
t "before it").
// rxq until next recvPkt (where it will h
i
t "before it").
if
c
.
rxqRead
.
Get
()
==
0
{
break
}
...
...
@@ -821,7 +821,7 @@ func (nl *NodeLink) serveRecv() {
// on rxq/acceptq only being put into the runqueue of current proc.
// By default proc runq will execute only when sendRecv blocks
// again next time deep in nl.recvPkt(), but let's force the switch
// now without additional wating to reduce latency.
// now without additional wa
i
ting to reduce latency.
// XXX bad - puts serveRecv to global runq thus with high p to switch M
//runtime.Gosched()
...
...
@@ -1602,7 +1602,7 @@ func (c *Conn) Ask(req Msg, resp Msg) error {
}
// ---- exchange of 1-1 request-reply ----
// (impedance matcher for current neo/py imlementation)
// (impedance matcher for current neo/py im
p
lementation)
// lightClose closes light connection.
//
...
...
go/neo/server/cluster_test.go
View file @
a00572f6
...
...
@@ -514,7 +514,7 @@ func TestMasterStorage(t *testing.T) {
// C loads every other {<,=}serial:oid - established link is reused
ziter
:=
zstor
.
Iterate
(
0
,
zodb
.
TidMax
)
ziter
:=
zstor
.
Iterate
(
bg
,
0
,
zodb
.
TidMax
)
// XXX hack: disable tracing early so that C.Load() calls do not deadlock
// TODO refactor cluster creation into func
...
...
go/zodb/cache.go
View file @
a00572f6
...
...
@@ -56,7 +56,6 @@ type Cache struct {
gcMu
sync
.
Mutex
lru
lruHead
// revCacheEntries in LRU order
size
int
// cached data size in bytes
sizeMax
int
// cache is allowed to occupy not more than this
}
...
...
@@ -163,8 +162,6 @@ func (c *Cache) Load(ctx context.Context, xid Xid) (buf *mem.Buf, serial Tid, er
// rce is not in cache - this goroutine becomes responsible for loading it
}
else
{
// XXX use connection poll
// XXX or it should be cared by loader?
c
.
loadRCE
(
ctx
,
rce
)
}
...
...
@@ -192,7 +189,6 @@ func (c *Cache) Prefetch(ctx context.Context, xid Xid) {
// spawn loading in the background if rce was not yet loaded
if
rceNew
{
// XXX use connection poll
go
c
.
loadRCE
(
ctx
,
rce
)
}
...
...
@@ -362,7 +358,7 @@ func (c *Cache) loadRCE(ctx context.Context, rce *revCacheEntry) {
// and will update lru and cache size for it itself.
rceOrig
:=
rce
rceDropped
:=
false
if
i
+
1
<
len
(
oce
.
rcev
)
{
if
i
+
1
<
len
(
oce
.
rcev
)
{
rceNext
:=
oce
.
rcev
[
i
+
1
]
if
rceNext
.
loaded
()
&&
tryMerge
(
rce
,
rceNext
,
rce
)
{
// not δsize -= len(rce.buf.Data)
...
...
@@ -544,7 +540,7 @@ func (c *Cache) gc() {
}
// freelist(OCE)
var
ocePool
=
sync
.
Pool
{
New
:
func
()
interface
{}
{
return
&
oidCacheEntry
{}
}
}
var
ocePool
=
sync
.
Pool
{
New
:
func
()
interface
{}
{
return
&
oidCacheEntry
{}
}}
// oceAlloc allocates oidCacheEntry from freelist.
func
oceAlloc
(
oid
Oid
)
*
oidCacheEntry
{
...
...
@@ -663,7 +659,7 @@ func (h *lruHead) rceFromInLRU() (rce *revCacheEntry) {
// errDB returns error about database being inconsistent
func
errDB
(
oid
Oid
,
format
string
,
argv
...
interface
{})
error
{
// XXX -> separate type?
return
fmt
.
Errorf
(
"cache: database inconsistency: oid: %v: "
+
format
,
return
fmt
.
Errorf
(
"cache: database inconsistency: oid: %v: "
+
format
,
append
([]
interface
{}{
oid
},
argv
...
)
...
)
}
...
...
go/zodb/cache_test.go
View file @
a00572f6
...
...
@@ -466,7 +466,7 @@ func TestCache(t *testing.T) {
fmt
.
Fprintf
(
bad
,
"unexpected rce found:
\n
%s
\n
"
,
pretty
.
Compare
(
expect
,
rce
))
}
if
bad
.
Len
()
!=
0
{
if
bad
.
Len
()
!=
0
{
t
.
Fatalf
(
"lookupRCE(%v):
\n
%s"
,
xid
,
bad
.
Bytes
())
}
}
...
...
@@ -637,7 +637,7 @@ func (c *Checker) assertEq(a, b interface{}) {
// ----------------------------------------
// noopStorage is dummy StorLoader which for any oid/xid always returns 1-byte data
type
noopStorage
struct
{}
type
noopStorage
struct
{}
var
noopData
=
[]
byte
{
0
}
func
(
s
*
noopStorage
)
URL
()
string
{
...
...
@@ -728,9 +728,9 @@ func benchLoadPar(b *testing.B, l StorLoader, worksize int) {
np
:=
runtime
.
GOMAXPROCS
(
0
)
p
:=
uint64
(
0
)
b
.
RunParallel
(
func
(
pb
*
testing
.
PB
)
{
b
.
RunParallel
(
func
(
pb
*
testing
.
PB
)
{
oid0
:=
Oid
(
atomic
.
AddUint64
(
&
p
,
+
1
))
// all workers start/iterate at different oid
xid
:=
Xid
{
At
:
1
,
Oid
:
oid0
}
xid
:=
Xid
{
At
:
1
,
Oid
:
oid0
}
for
pb
.
Next
()
{
buf
,
_
,
err
:=
l
.
Load
(
ctx
,
xid
)
if
err
!=
nil
{
...
...
@@ -797,7 +797,7 @@ func benchLoadProc(pb *testing.PB, l StorLoader, worksize int) error {
}
func
BenchmarkNoopStorageProc
(
b
*
testing
.
B
)
{
b
.
RunParallel
(
func
(
pb
*
testing
.
PB
)
{
b
.
RunParallel
(
func
(
pb
*
testing
.
PB
)
{
s
:=
&
noopStorage
{}
err
:=
benchLoadProc
(
pb
,
s
,
b
.
N
)
if
err
!=
nil
{
...
...
@@ -807,7 +807,7 @@ func BenchmarkNoopStorageProc(b *testing.B) {
}
func
BenchmarkCacheStartupProc
(
b
*
testing
.
B
)
{
b
.
RunParallel
(
func
(
pb
*
testing
.
PB
)
{
b
.
RunParallel
(
func
(
pb
*
testing
.
PB
)
{
s
:=
&
noopStorage
{}
c
:=
NewCache
(
s
,
b
.
N
)
err
:=
benchLoadProc
(
pb
,
c
,
b
.
N
)
...
...
@@ -821,7 +821,7 @@ func BenchmarkCacheStartupProc(b *testing.B) {
func
benchEachCacheProc
(
b
*
testing
.
B
,
f
func
(
b
*
testing
.
B
,
pb
*
testing
.
PB
,
c
*
Cache
)
error
)
{
for
_
,
size
:=
range
cachesizev
{
b
.
Run
(
fmt
.
Sprintf
(
"size=%d"
,
size
),
func
(
b
*
testing
.
B
)
{
b
.
RunParallel
(
func
(
pb
*
testing
.
PB
)
{
b
.
RunParallel
(
func
(
pb
*
testing
.
PB
)
{
s
:=
&
noopStorage
{}
c
:=
NewCache
(
s
,
size
)
err
:=
f
(
b
,
pb
,
c
)
...
...
go/zodb/storage/fs1/filestorage.go
View file @
a00572f6
...
...
@@ -52,7 +52,7 @@
// https://github.com/zopefoundation/ZODB/blob/a89485c1/src/ZODB/fsIndex.py
// https://github.com/zopefoundation/ZODB/commit/1bb14faf
//
// Unless one is doing something FileStorage-specific, it is advi
c
ed not to use
// Unless one is doing something FileStorage-specific, it is advi
s
ed not to use
// fs1 package directly, and instead link-in lab.nexedi.com/kirr/neo/go/zodb/wks,
// open storage by zodb.OpenStorage and use it by way of zodb.IStorage interface.
//
...
...
@@ -131,7 +131,7 @@ func (fs *FileStorage) Load(_ context.Context, xid zodb.Xid) (buf *mem.Buf, seri
// FIXME zodb.TidMax is only 7fff... tid from outside can be ffff...
// -> TODO reject tid out of range
// FIXME kill Load_XXXWithNextSerial after neo/py cache does not depend on next_serial
// FIXME kill Load_XXXWithNextSerial
XXX
after neo/py cache does not depend on next_serial
buf
,
serial
,
_
,
err
=
fs
.
Load_XXXWithNextSerialXXX
(
nil
,
xid
)
return
buf
,
serial
,
err
}
...
...
@@ -385,7 +385,7 @@ func (fs *FileStorage) findTxnRecord(r io.ReaderAt, tid zodb.Tid) (TxnHeader, er
}
// Iterate creates zodb-level iterator for tidMin..tidMax range
func
(
fs
*
FileStorage
)
Iterate
(
tidMin
,
tidMax
zodb
.
Tid
)
zodb
.
ITxnIterator
{
func
(
fs
*
FileStorage
)
Iterate
(
_
context
.
Context
,
tidMin
,
tidMax
zodb
.
Tid
)
zodb
.
ITxnIterator
{
// when iterating use IO optimized for sequential access
fsSeq
:=
xbufio
.
NewSeqReaderAt
(
fs
.
file
)
ziter
:=
&
zIter
{
iter
:
Iter
{
R
:
fsSeq
}}
...
...
@@ -403,7 +403,7 @@ func (fs *FileStorage) Iterate(tidMin, tidMax zodb.Tid) zodb.ITxnIterator {
URL
:
fs
.
URL
(),
Op
:
"iter"
,
// XXX (?) add TidRange type which prints as
// "tidmin..tidmax" with omiting ends if it is either 0 or ∞
// "tidmin..tidmax" with omit
t
ing ends if it is either 0 or ∞
Args
:
[]
zodb
.
Tid
{
tidMin
,
tidMax
},
Err
:
err
,
}}
...
...
go/zodb/storage/fs1/filestorage_test.go
View file @
a00572f6
...
...
@@ -157,7 +157,7 @@ func TestLoad(t *testing.T) {
// iterate tidMin..tidMax and expect db entries in expectv
func
testIterate
(
t
*
testing
.
T
,
fs
*
FileStorage
,
tidMin
,
tidMax
zodb
.
Tid
,
expectv
[]
dbEntry
)
{
ctx
:=
context
.
Background
()
iter
:=
fs
.
Iterate
(
tidMin
,
tidMax
)
iter
:=
fs
.
Iterate
(
ctx
,
tidMin
,
tidMax
)
fsi
,
ok
:=
iter
.
(
*
zIter
)
if
!
ok
{
_
,
_
,
err
:=
iter
.
NextTxn
(
ctx
)
...
...
@@ -233,7 +233,7 @@ func testIterate(t *testing.T, fs *FileStorage, tidMin, tidMax zodb.Tid, expectv
txe
:=
dbe
.
Entryv
[
kdata
]
dh
:=
txe
.
Header
// assert datai point
e
s to where we expect - this will allow us
// assert datai points to where we expect - this will allow us
// not only to check oid/tid/data but also to check whole data header.
if
datai
!=
&
fsi
.
datai
{
t
.
Fatal
(
"unexpected datai pointer"
)
...
...
@@ -309,7 +309,7 @@ func BenchmarkIterate(b *testing.B) {
b
.
ResetTimer
()
for
i
:=
0
;
i
<
b
.
N
;
i
++
{
iter
:=
fs
.
Iterate
(
zodb
.
Tid
(
0
),
zodb
.
TidMax
)
iter
:=
fs
.
Iterate
(
ctx
,
zodb
.
Tid
(
0
),
zodb
.
TidMax
)
for
{
txni
,
dataIter
,
err
:=
iter
.
NextTxn
(
ctx
)
...
...
go/zodb/str.go
View file @
a00572f6
...
...
@@ -21,9 +21,9 @@ package zodb
// formatting and parsing for basic zodb types
import
(
"fmt"
"encoding/hex"
"encoding/binary"
"encoding/hex"
"fmt"
"lab.nexedi.com/kirr/go123/xfmt"
"lab.nexedi.com/kirr/go123/xstrings"
...
...
@@ -87,7 +87,7 @@ func (xid Xid) XFmtString(b xfmt.Buffer) xfmt.Buffer {
func
parseHex64
(
subj
,
s
string
)
(
uint64
,
error
)
{
// XXX -> xfmt ?
// XXX like scanf("%016x") but scanf implicitly skips spaces without giving control to caller and is slower
var
b
[
8
]
byte
var
b
[
8
]
byte
if
len
(
s
)
!=
16
{
return
0
,
fmt
.
Errorf
(
"%s %q invalid"
,
subj
,
s
)
}
...
...
go/zodb/zodb.go
View file @
a00572f6
...
...
@@ -88,8 +88,7 @@ type TxnInfo struct {
Extension
[]
byte
}
// DataInfo represents information about one object change.
// DataInfo is information about one object change.
type
DataInfo
struct
{
Oid
Oid
Tid
Tid
// changed by this transaction
...
...
@@ -152,7 +151,7 @@ type OpError struct {
URL
string
// URL of the storage
Op
string
// operation that failed
Args
interface
{}
// operation arguments, if any
Err
error
// actual error that occured during the operation
Err
error
// actual error that occur
r
ed during the operation
}
func
(
e
*
OpError
)
Error
()
string
{
...
...
@@ -169,7 +168,6 @@ func (e *OpError) Cause() error {
}
// IStorage is the interface provided by opened ZODB storage
type
IStorage
interface
{
IStorageDriver
...
...
@@ -256,8 +254,11 @@ type IStorageDriver interface {
// Iterate creates iterator to iterate storage in [tidMin, tidMax] range.
//
// Iterate does not return any error. If there was error when setting
// iteration up - it will be returned on first NextTxn call.
//
// TODO allow iteration both ways (forward & backward)
Iterate
(
tidMin
,
tidMax
Tid
)
ITxnIterator
// XXX ctx , error ?
Iterate
(
ctx
context
.
Context
,
tidMin
,
tidMax
Tid
)
ITxnIterator
}
// ITxnIterator is the interface to iterate transactions.
...
...
go/zodb/zodbtools/catobj.go
View file @
a00572f6
...
...
@@ -58,7 +58,6 @@ func Dumpobj(ctx context.Context, w io.Writer, stor zodb.IStorage, xid zodb.Xid,
return
err
}
// XXX hack - TODO rework IStorage.Load to fill-in objInfo directly?
objInfo
.
Oid
=
xid
.
Oid
objInfo
.
Tid
=
serial
objInfo
.
Data
=
buf
.
Data
...
...
go/zodb/zodbtools/dump.go
View file @
a00572f6
...
...
@@ -193,7 +193,7 @@ func (d *dumper) Dump(ctx context.Context, stor zodb.IStorage, tidMin, tidMax zo
var
dataIter
zodb
.
IDataIterator
var
err
error
iter
:=
stor
.
Iterate
(
tidMin
,
tidMax
)
iter
:=
stor
.
Iterate
(
ctx
,
tidMin
,
tidMax
)
// transactions
for
{
...
...
go/zodb/zodbtools/dump_test.go
View file @
a00572f6
...
...
@@ -45,7 +45,7 @@ func loadZdumpPy(t *testing.T, path string) string {
t
.
Fatal
(
err
)
}
// python q
ou
tes "\v" as "\x0b", go as "\v"; same for "\f", "\a", "\b".
// python q
uo
tes "\v" as "\x0b", go as "\v"; same for "\f", "\a", "\b".
// XXX this is a bit hacky. We could compare quoted strings as decoded,
// but this would need zdump format parser which could contain other
// bugs. Here we want to compare output ideally bit-to-bit but those
...
...
go/zodb/zodbtools/info.go
View file @
a00572f6
...
...
@@ -126,6 +126,7 @@ func infoMain(argv []string) {
if
err
!=
nil
{
prog
.
Fatal
(
err
)
}
// TODO defer stor.Close()
err
=
Info
(
ctx
,
os
.
Stdout
,
stor
,
argv
[
1
:
])
if
err
!=
nil
{
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment