1. 20 Mar, 2005 3 commits
    • Chris McDonough's avatar
      No commit message · 46014ce7
      Chris McDonough authored
      No commit message
      46014ce7
    • Chris McDonough's avatar
      Add a record iteration protocol to FileStorage. You can use the record... · 7f220b80
      Chris McDonough authored
      Add a record iteration protocol to FileStorage.  You can use the record iterator to iterate over all current revisions
      of data pickles in the storage.
      
      In order to support calling via ZEO, we don't implement this as an actual iterator.  An example of using the record iterator
      protocol is as follows:
      
      storage = FileStorage('anexisting.fs')
      next_oid = None
      while 1:
          oid, tid, data, next_oid = storage.record_iternext(next_oid)
          # do something with oid, tid and data
          if next_oid is None:
              break
      
      The behavior of the iteration protocol is now to iterate over all current records in the database in ascending oid order, 
      although this is not a promise to do so in the future.
      
      
      7f220b80
    • Tim Peters's avatar
      Merge rev 29592 from 3.3 branch. · ac5116e4
      Tim Peters authored
      Document exclude{min,max}.
      
      The ZODB4 BTrees excludemin/excludemax arguments got implemented
      in 3.3, but the corresponding changes to the BTrees interface
      file got dropped on the floor.
      ac5116e4
  2. 17 Mar, 2005 1 commit
    • Tim Peters's avatar
      Merge rev 29532 from 3.3 branch. · 8218f4bb
      Tim Peters authored
      Merge checkin made from a wrong project:
      
      r29511 | gotcha | 2005-03-16 11:01:21 -0500 (Wed, 16 Mar 2005) | 1 line
      Changed paths:
         M /Zope/trunk/utilities/ZODBTools/zodbload.py
      
      port features from 2.7
      8218f4bb
  3. 14 Mar, 2005 1 commit
    • Tim Peters's avatar
      Merge rev 29467 from 3.3 branch. · d7276762
      Tim Peters authored
      Backward compatibility hack for ZODB.Persistent{List,Mapping}.
      
      Allow those old (ZODB 3.2) dotted paths to work again.
      
      Fallout:  testPersistentMapping.py hasn't actually run anything
      for a long time, due to a mysterious "return None" at the start
      of its test_suite() function.  Removed that.  Then its
      checkNewPicklesAreSafe test failed.  Spent 15 minutes on that,
      and gave up -- I'm still not even sure what it's _trying_ to
      test.  Changed it to a TODO.
      d7276762
  4. 11 Mar, 2005 3 commits
  5. 07 Mar, 2005 1 commit
  6. 03 Mar, 2005 1 commit
    • Tim Peters's avatar
      Merge ZODB trunk changes checked in from a wrong project. · 7c665d9a
      Tim Peters authored
      r29290 | frerich | 2005-02-24 17:36:00 -0500 (Thu, 24 Feb 2005)
      Changed paths:
         M /Zope3/trunk/src/ZODB/tests/dbopen.txt
         ...
         minor editing
      
      r29247 | gintautasm | 2005-02-22 06:40:26 -0500 (Tue, 22 Feb 2005)
      Changed paths:
         M /Zope3/trunk/src/BTrees/Interfaces.py
         ...
         More minor nitpicks.  This should be the last one.
      
      7c665d9a
  7. 02 Mar, 2005 3 commits
  8. 01 Mar, 2005 1 commit
  9. 28 Feb, 2005 1 commit
    • Tim Peters's avatar
      Merge rev 29350 from ZODB 3.3 branch. · e6f505df
      Tim Peters authored
      Port from ZODB 3.2.
      
      Change FileStorage .restore() and .store() to update max oid in use.
      
      This is the last of the checkins to fix critical bugs involving rare cases
      where a FileStorage could end up reusing old oids for new objects.
      e6f505df
  10. 25 Feb, 2005 1 commit
    • Tim Peters's avatar
      Merge rev 29302 from 3.3 branch. · ed9ebe85
      Tim Peters authored
      Port from ZODB 3.2.
      
      Stop believing the maximum oid cached in a FileStorage's .index file.
      
      This is a critical bugfix, although the problems it addresses are
      (a) rare; and, (b) not entirely fixed yet (more checkins to come).
      
      The true max oid is found efficiently now by exploiting the recently-added
      fsIndex.maxKey() method (which was, of course, added for this purpose).
      
      Also fix that the .index file could get updated on disk when the
      FileStorage was opened in read-only mode.  The code was trying to prevent
      this, but missed the most obvious rewrite path.
      
      Incidentally improved many obsolete and/or incorrect comments.
      ed9ebe85
  11. 24 Feb, 2005 2 commits
    • Tim Peters's avatar
      Merge rev 29291 from ZODB 3.3 branch. · f82de536
      Tim Peters authored
      Port from ZODB 3.2.
      
      Give fsIndex an efficient maxKey() implementation.
      
      This will (in a later checkin) be used to give FileStorage an "obviously
      correct" way to determine the largest oid used in an .fs.index file.
      f82de536
    • Tim Peters's avatar
      Merge rev 29286 from ZODB 3.3 branch. · f2a5c8f4
      Tim Peters authored
      Port from ZODB 3.2.
      
      Note that cPickleCache.c must be recompiled else the new test will fail.
      
      Change the exception raised when an attempt is made to add two objects to
      the cache with the same oid.  The former messsage didn't make sense.
      
      Add a test to verify that this exception does get raised, and that the
      message given is the intended one.
      
      This is the first of a series of checkins, to fix critical bugs where
      ZODB can in fact raise this exception in rare, but normal, use cases.
      f2a5c8f4
  12. 17 Feb, 2005 1 commit
    • Tim Peters's avatar
      Merge rev 29194 from 3.3 branch. · 138bb41f
      Tim Peters authored
      setUp():  Use a much smaller (than default) ZEO client cache.
      
      The current default is 200MB, and at least testSerialization reads
      the entire cache file into one giant string.  That grossly boosts
      the process highwater mark when running the tests.  Windows reclaims
      the memory after, but other platforms may not.
      
      Cool:  testSerialization was one of the slowest tests before, but
      is 100x faster now.
      138bb41f
  13. 10 Feb, 2005 1 commit
    • Tim Peters's avatar
      Merge rev 29107 from 3.3 branch. · 2964cca6
      Tim Peters authored
      Side-port changes checked in from a wrong project (Zope trunk):
      
      r28991 | jim | 2005-01-31 10:45:46 -0500 (Mon, 31 Jan 2005) | 5 lines
      ...
      Renamed the Zope package to Zope2
      
      Import of the Zope package is now deprecated, but will
      be supported until Zope 2.11.
      2964cca6
  14. 09 Feb, 2005 1 commit
    • Tim Peters's avatar
      Port rev 29092 from 3.3 branch. · 7a473f11
      Tim Peters authored
      Forward port from ZODB 3.2.
      
      Connection.__init__():  Python 2.4 added a new gimmick to asyncore (a
      ._map attribute on asyncore.dispatcher instances) that breaks the
      delicate ZEO startup dance.  Repaired that.
      7a473f11
  15. 05 Feb, 2005 1 commit
    • Tim Peters's avatar
      Merge rev 29052 from 3.3 branch. · 3ebdc9a2
      Tim Peters authored
      Port from ZODB 3.2.
      
      Fixed several thread and asyncore races in ZEO's connection dance.
      
      ZEO/tests/ConnectionTests.py
          The pollUp() and pollDown() methods were pure busy loops whenever
          the asyncore socket map was empty, and at least on some flavors of
          Linux that starved the other thread(s) trying to do real work.
          This grossly increased the time needed to run tests using these, and
          sometimes caused bogus "timed out" test failures.
      
      ZEO/zrpc/client.py
      ZEO/zrpc/connection.py
          Renamed class ManagedConnection to ManagedClientConnection, for clarity.
      
          Moved the comment block about protocol negotiation from the guts of
          ManagedClientConnection to before the Connection base class -- the
          Connection constructor can't be understood without this context.  Added
          more words about the delicate protocol negotiation dance.
      
          Connection class:  made this an abstract base clase.  Derived classes
          _must_ implement the handshake() method.  There was really nothing in
          common between server and client wrt what handshake() needs to do, and
          it was confusing for one of them to use the base class handshake() while
          the other replaced handshake() completely.
      
          Connection.__init__:  It isn't safe to register with asyncore's socket
          map before special-casing for the first (protocol handshake) message is
          set up.  Repaired that.  Also removed the pointless "optionalness" of
          the optional arguments.
      
          ManagedClientConnection.__init__:  Added machinery to set up correct
          (thread-safe) message queueing.  There was an unrepairable hole before,
          in the transition between "I'm queueing msgs waiting for the server
          handshake" and "I'm done queueing messages":  it was impossible to know
          whether any calls to the client's "queue a message" method were in
          progress (in other threads), so impossible to make the transition safely
          in all cases.  The client had to grow its own message_output() method,
          with a mutex protecting the transition from thread races.
      
          Changed zrpc-conn log messages to include "(S)" for server-side or
          "(C)" for client-side.  This is especially helpful for figuring out
          logs produced while running the test suite (the server and client
          log messages end up in the same file then).
      3ebdc9a2
  16. 04 Feb, 2005 2 commits
  17. 03 Feb, 2005 3 commits
  18. 24 Jan, 2005 2 commits
  19. 12 Jan, 2005 2 commits
  20. 11 Jan, 2005 2 commits
  21. 07 Jan, 2005 1 commit
    • Tim Peters's avatar
      Merge rev 28769 from 3.3 branch. · 75fe517f
      Tim Peters authored
      Forward-port from ZODB 3.2.
      
      Collector 1503:  excessive logging.
      
      ClientStorage._wait_sync():  Don't log more than one "waiting for cache
      verification to finish" message per 5 minutes.
      75fe517f
  22. 23 Dec, 2004 1 commit
    • Tim Peters's avatar
      Merge rev 28701 from 3.3 branch. · 743ef117
      Tim Peters authored
      Critical bugfix for invalidations during ZEO cache verification.
      
      ClientCache.invalidate():
      
      - This was written not to expect tid=None, which is what happens
        during cache verification, although it didn't catch the tid=None
        case and went on doing really strange things then.  Rewrote so
        that tid=None means "forget everything I think I know about
        the oid".  It was kind-of performing an invalidation wrt the
        in-memory data structures in this case, but leaving the
        invalidated data for oid looking as if it were still current on
        disk.  That was accidental, depending on what various other
        parts of the code thought "None" meant (primarily, an end_tid
        of None is generally taken here to mean "the data is current").
        
      - Also changed the "and we already have version data for oid" case
        to a "forget everything I think I know about the oid" case.  The
        comments seemed to say that's what it thought it was doing, but
        the code actually left non-current revisions sitting around.  I
        really don't want to spend time trying to figure out what versions
        are supposed to do in all cases here, and nuking everything in
        this case is an obviously *safe* thing to do.
      
      - Added more asserts, because the pre- and post-conditions here are
        delicate and non-obvious (indeed, that a new assert failed is how
        I stumbled into the invalidation problem to begin with).
      
      FileCache.remove():  Changed to remove the key from self.key2entry.
      While self.filemap keeps an in-memory map of the entire disk file,
      there's no need to keep info in key2entry for keys that no longer
      exist in the disk file.  I'm not sure a visible bug followed from
      this (but not sure one didn't either!), but it at best wasted
      memory, and violated a documented invariant.
      
      ConnectionTests.py:  New test checkVerificationInvalidationPersists().
      Before the patch, this provokes a persistent ZEO cache into
      delivering stale object state.  Of course the test passes after
      this patch.
      
      stats.py:  Added a trace code meaning "invalidation hit, and the
      object state was discarded".  This is what happens now for
      invalidations produced by verification.  Because of MVCC, that's
      really not needed *except* for the invalidations produced by
      startup verification (invalidations normally just set a new
      ending tid on the validity range for what was current data; but
      validity-range info isn't available from verification invalidations).
      
      743ef117
  23. 22 Dec, 2004 1 commit
    • Tim Peters's avatar
      Merge rev 28685 from 3.3 branch. · 2855b2d9
      Tim Peters authored
      Collector 1536:  ClientStorage.__init__ ignores cache_size.
      
      Also split test checkRollover into two tests,
      checkDisconnectedCacheWorks (persistent ZEO cache large enough
      to satisfy all requests) and checkDisconnectedCacheFails (cache
      too small to satisfy all requests while disconnected).
      2855b2d9
  24. 21 Dec, 2004 4 commits
    • Tim Peters's avatar
      Merge rev 28680 from 3.3 branch. · dc518849
      Tim Peters authored
      More cleanup, and a new problem.  Notable:
      
      Object.serialize_header():  Stopped stuffing the version string into this
      notion of "a header".  Couldn't see any reason for it, and it was
      surprising.
      
      ClientCache.invalidate():  Added some asserts to clarify intended
      preconditions.  Alas, my guess that it was intended that the passed-in
      tid always be greater than the current tid turned out to trigger
      errors.  This gets complicated, and hasn't been resolved yet.  The
      problem is that the passed-in tid can be (in fact, always is) None
      during cache verification.  The code here *really* doesn't seem to
      be expecting that, and the on-disk cache file is clearly left in a
      wrong state (if we repopulated the cache from the disk file, the
      object would not look invalidated any more, it would look current).
      Dumped in a giant stack trace so I don't forget this; sent email to
      Jeremy asking if he recalls what the real intent was.
      dc518849
    • Tim Peters's avatar
      Merge rev 28673 from 3.3 branch. · 60e508aa
      Tim Peters authored
      More cleanup.  Notable:
      
      - Object.fromFile():  renamed `header_only` arg to `skip_data`.
        Skipping the data is what it does, while there are at least 3
        distinct notions of what "a header" means in this module.
        Object has two notions of "header" all by itself (& I'm still
        not sure why Object.serialize_header() considers the version
        string to be part of "the header").
      
      - FileCache.__init__():  log a warning if reuse=True but the
        given file path doesn't exist.  The code ignores `reuse` then
        (before, and now).  Not sure that's the best thing to do.
      60e508aa
    • Tim Peters's avatar
      Merged rev 28671 from 3.3 branch. · 6f12261d
      Tim Peters authored
      More cleanup.  Notable:
      
      - Moved the logger object to module scope.  There's virtually
        no logging of any kind done here, and I suspect that's partly
        because the logging object was clumsy to get at.
      
      - Figured out what the code actually does when the cache size
        asked for doesn't match the actual size of a pre-existing
        cache file.  It apparently wanted to ignore the requested
        size then.  It probably shouldn't, but leaving that for later.
        Still, it left its own idea of the file size out of synch
        with the actual file size, and that was a bug.  For now,
        logged a warning when this happens, and changed the code to
        respect the actual file size.
      6f12261d
    • Tim Peters's avatar
      Merged rev 28664 from 3.3 branch. · 10324308
      Tim Peters authored
      More cleanup.  Notable:
      
      ClientCache:  Removed the .tid attribute.  It was never updated; the
      contained FileCache instance actually keeps track of the last tid.
      This had the happy side effect of fixing bugs in testSerialization:
      it was accessing .tid directly, and so was _always_ comparing None to
      None.  Changing the test to use .getLastTid() instead means it's now
      testing what it always intended to test.
      
      FileCache:  Removed some unused private attributes for tracking
      internal statistics.
      10324308