Commit 4cfe3662 authored by Jim Fulton's avatar Jim Fulton

refinements (including spelling and considering allowing tpc_finish and loads...

refinements (including spelling and considering allowing tpc_finish and loads to be in flight at once.
parent b470bb65
...@@ -2,15 +2,15 @@ ...@@ -2,15 +2,15 @@
Response ordering requirements Response ordering requirements
============================== ==============================
ZEO servers are logically concurrent because they server multiple ZEO servers are logically concurrent because they serve multiple
clients in multiple threads. Because of this, we have to make sure clients in multiple threads. Because of this, we have to make sure
that information about object histories remain consistent. that information about object histories remain consistent.
An object history is a sequence of object revisions. Each revision has An object history is a sequence of object revisions. Each revision has
a tid, which is essentially a time stamp. a tid, which is essentially a time stamp.
We load objects using either load, which retirns the current We load objects using either ``load``, which returns the current
object. or loadBefore, which returns the object before a specic time/tid. object. or loadBefore, which returns the object before a specific time/tid.
When we cache revisions, we record the tid and the next/end tid, which When we cache revisions, we record the tid and the next/end tid, which
may be None. The end tid is important for choosing a revision for may be None. The end tid is important for choosing a revision for
...@@ -19,15 +19,15 @@ current, for load. ...@@ -19,15 +19,15 @@ current, for load.
Because the client and server are multi-threaded, the client may see Because the client and server are multi-threaded, the client may see
data out of order. Let's consider some scenarios. In these data out of order. Let's consider some scenarios. In these
scenarious, we'll consider a single object with revisions t1, t2, etc. scenarios, we'll consider a single object with revisions t1, t2, etc.
We consider loading pretty generically, as bith load and loadBefore We consider loading pretty generically, as bath load and loadBefore
are similar in that they may have data about current revisions. are similar in that they may have data about current revisions.
Scenarios Scenarios
========= =========
S1 S1
Client sees load results before later invalidations Client sees load results before earlier invalidations
- server commits t1 - server commits t1
...@@ -42,9 +42,9 @@ S1 ...@@ -42,9 +42,9 @@ S1
- client gets invalidation for t2, client should ignore - client gets invalidation for t2, client should ignore
This scenario could occur because the commits are for a different This scenario could occur because the commits are for a different
client, and the server doesn't block loads while committing. This client, and a hypothetical; server doesn't block loads while
situation is pretty easy to deal with, as we just ignore committing. This situation is pretty easy to deal with, as we just
invalidations for earlier revisions. ignore invalidations for earlier revisions.
Note that invalidations will never come out of order from the server. Note that invalidations will never come out of order from the server.
...@@ -59,9 +59,9 @@ S2 ...@@ -59,9 +59,9 @@ S2
- Client receives t2 in load result. - Client receives t2 in load result.
- Client recieves t1 in tpc_finish result, doesn't invalidate anything - Client receives t1 in tpc_finish result, doesn't invalidate anything
- Client recieves t2 in tpc_finish result, doesn't invalidate anything - Client receives t2 in tpc_finish result, doesn't invalidate anything
This scenario is equivalent to S1. This scenario is equivalent to S1.
...@@ -72,47 +72,63 @@ S3 ...@@ -72,47 +72,63 @@ S3
- server commits t2 - server commits t2
- Client recieves invalidation for t2. - Client receives invalidation for t2.
- Client recieves load result for t1. - Client receives load result for t1.
This scenario id worrisome because the data that needs to be This scenario is worrisome because the data that needs to be
invalidated isn't present when the invalidation arrives. invalidated isn't present when the invalidation arrives.
S4 S4
Client sees commit resulrs before load results. Client sees commit results before load results.
- Client loads, storage reads t1. - Client loads, storage reads t1.
- Client commits, storage commits t2. - Client commits, storage commits t2.
- Client recieves t2 in tpc_finish result. - Client receives t2 in tpc_finish result.
- Client recieves load result for t1. - Client receives load result for t1.
This scenario is equivalent to S3. This scenario is equivalent to S3.
Implemenation notes Implementation notes
=================== ===================
First, it's worth noting that the server sends data to the client in
correct order with respect to loads and invalidations (or tpc_finish
results). This is a consequence of the fact that invalidations are
sent in a callback called when the storage lock is held, blocking
loadd while committing, and the fact that client requests, for a
particular client, are handled by a single thread on the server.
Invalidations are sent from different threads that clients. Outgoing
data is queued, however, using Python lists, which are protected by
the GIL. This means that the serialization provided though storage
locks is preserved by the way that server outputs are queued.
ZEO 4 ZEO 4
----- -----
In ZEO 4, invalidations and loads are handled by separate
threads. This means that even though data arive in order, they may not
be processed in order,
S1 S1
The existing servers mitigates this by blocking loads while The existing servers mitigate this by blocking loads while
committing. Because this is easy to deal with, maybe in the future committing. On the client, this is still a potential issue because loads
the server will allow loads to be done before invalidations are
sent. On the client, this is still a potential issue because loads
and invalidations are handled by separate threads. and invalidations are handled by separate threads.
The client is conservative because it always forgets current data in The client cache is conservative because it always forgets current data in
memory when it sees an invalidation data for an object. memory when it sees an invalidation data for an object.
The client gets the scenario wrong because it checks for The client gets this scenario wrong, in an edge case, because it
invalidations matching the current tid, but not invalidations before checks for invalidations matching the current tid, but not
the current tid. If the thread handling invalidations was slow invalidations before the current tid. If the thread handling
enough for this scenario to occur, then the cache would end up with invalidations was slow enough for this scenario to occur, then the
an end tid < a starting tid. This is probably very unlikely. cache would end up with an end tid < a starting tid. This is
probably very unlikely.
S2 S2
The existing clients prevent this by serializing commits with each The existing clients prevent this by serializing commits with each
...@@ -133,30 +149,34 @@ S4 ...@@ -133,30 +149,34 @@ S4
As with S2, clients mitigate this by preventing simultaneous loads As with S2, clients mitigate this by preventing simultaneous loads
and commits. and commits.
Note that on the server, invalidations are sent from different threads
that clients. Outgoing data is queued, however, using Python lists,
which are protected by the GIL. This means that the serialization
provided though storage locks is preserved by the way that server
outputs are queued.
ZEO 5 ZEO 5
----- -----
In ZEO(/ZODB) 5, we want to get more conccurency, both on the client, In ZEO(/ZODB) 5, we want to get more concurrency, both on the client,
and on the server. On the client, cache invalidations and loads are and on the server. On the client, cache invalidations and loads are
done by the same thread, which makes things a bit simpler. This let's done by the same thread, which makes things a bit simpler. This let's
us get rid of the client load lock and prevents the scenarios above us get rid of the client load lock and prevents the scenarios above
with existing servers and storages. with existing servers and storages.
Going forward, however, we'd like to allow greater server On the client, we'd like to stop serializing loads and commits. We'd
concurrency. The main opportunity here is allowing commits for like commits (tpc_finish calls) to in flight with loads (and with
separate oids to happen concurrently. This wouldn't effect the other commits). In the current protocol, tpc_finish, load and
invalidation/load ordering though, assuming we blocked loading an oid loadBefore are all synchronous calls that are handled by a single
while it was being committed in tpc_finish. thread on the server, so these calls end up being serialized on the
server.
If we ever allowed multiple threads to service client requests, then
we'd need to consider scenario S4, but this isn't an issue now (or for
the foreseeable future).
The main server opportunity is allowing commits for separate oids to
happen concurrently. This wouldn't effect the invalidation/load
ordering though, assuming we continued to block loading an oid while
it was being committed in tpc_finish.
We could also allow loads to proceed while invalidations are being We could also allow loads to proceed while invalidations are being
queued for an object. Queuing invalidations is pretty fast though. It's queued for an object. Queuing invalidations is pretty fast though. It's
not clear that this would be much of a win. This probaby isn't worth not clear that this would be much of a win. This probably isn't worth
fooling with for now. If we did want to relax this, we could, on the fooling with for now. If we did want to relax this, we could, on the
client, track invalidations for outstanding load requests and adjust client, track invalidations for outstanding load requests and adjust
how we wrote data to the cache accordingly. Again, we won't bother in how we wrote data to the cache accordingly. Again, we won't bother in
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment