1. 15 Feb, 2018 1 commit
    • Jim Fulton's avatar
      Avoid seeking on import (#188) · 2115c90a
      Jim Fulton authored
      * Don't seek on imports
      
      (other than the possible seek for custom importers)
      
      We were seeking to handle blob markers. This has two major drawbacks:
      
      1. It wasn't possible to use a non-seekable file.  A use case for
         export/import is to copy database data.  An intermediate file, and
         associated I/O, could be avoided using a pipe, but pipes aren't
         seekable.
      
      2. Seeks cause file-buffer data to be discarded, making IO far more
         expensive.
      
      We didn't really need blob markers, because the preceeding blob data
      records serve as markers.  (Now we're stuck with them for backward
      compatibility.)
      
      * Make cp's buffer size larger and configurable.
      
      * Use the storage temprary directory when importing blobs
      
      To avoid an extra copy.
      
      Also, allow the copy (cp) buffer sie to be overridden on export.
      
      (I see no obvious way to plumb it on import. :( )
      
      * Oops, need to use a binary literal (Python 3)
      
      * Respond to PR comments
      2115c90a
  2. 14 Nov, 2017 1 commit
    • Jim Fulton's avatar
      Update the front page to be a little less negative. (#186) · de1f24ca
      Jim Fulton authored
      Especially with regard to write capacity and search.
      
      This hasn't been updated in a while and I feel that it is out of date.
      
      I feel that this could use more work, but I'm anxious to do an
      incremental improvement given the screencast I just released. :)
      de1f24ca
  3. 13 Nov, 2017 2 commits
  4. 12 Nov, 2017 1 commit
  5. 08 Nov, 2017 1 commit
    • Julien Muchembled's avatar
      Review Connection/DB closure · 1b9475d4
      Julien Muchembled authored
      In addition to some micro-optimisation, this fixes the following minor issues:
      - Closing a DB left Connections registered to the global transaction manager.
        Which broke at least multi-db because only such primary connections were
        ignored with monkey-patches.
      - Stop calling the real newTransaction when aborting an existing transaction.
        This was not the case if explicit_transactions=0.
      1b9475d4
  6. 07 Nov, 2017 1 commit
  7. 30 Oct, 2017 1 commit
  8. 27 Oct, 2017 2 commits
  9. 05 Oct, 2017 1 commit
  10. 30 Aug, 2017 2 commits
  11. 29 Aug, 2017 2 commits
    • Jason Madden's avatar
      Use a higher pickle protocol for serializing objects on Python 2 (#179) · be5a9d54
      Jason Madden authored
      * Use a higher pickle protocol (2) for serializing objects on Python 2
      
      Previously protocol 1 was used. This is more efficient for new-style
      classes (all persistent objects are new-style), according to the docs,
      at the cost of being very slightly less space efficient for old-style
      classes.
      
      In tests of a persistent object with two trivial numeric attributes,
      the higher protocol was 12 bytes smaller, and serialized and
      deserialized 1us faster. Introducing a reference to another new-style
      class for a more realistic test made the higher protocol twice as fast
      to serialize (20.5 vs 10.3us), almost half the size (215 vs 142
      bytes), and it deserialized 30% faster (6.5 vs 4.6us).
      
      On Python 2, this will now allow open ``file`` objects to be
      pickled (loading the object will result in a closed file); previously
      this would result in a ``TypeError`` (as does under Python 3). We had
      tests that you couldn't do that with a BlobFile so I had to update it
      to still make that true.
      
      I wouldn't recommend serializing arbitrary open files under Python
      2 (for one thing, they can't trivially be deserialized in Python 3),
      but I didn't take any steps to prevent it either. Since this hasn't
      been possible, there shouldn't be code in the wild that is trying to
      do it---and it wouldn't be forward compatible with Python 3 either.
      be5a9d54
    • XeL64's avatar
      fix typo (#180) · b340b651
      XeL64 authored
      b340b651
  12. 25 Jul, 2017 4 commits
  13. 24 Jul, 2017 11 commits
  14. 17 May, 2017 5 commits
  15. 17 Apr, 2017 1 commit
    • Jason Madden's avatar
      Make all classes new-style. (#160) · e714f515
      Jason Madden authored
      * Make all classes new-style.
      
      On PyPy, it's documented (http://pypy.org/performance.html#id3) that
      old-style classes are slow, and classes that derive from both old and
      new are especially slow. I checked with the PyPy devs on IRC today and
      they confirmed that's still true with the latest PyPy2 releases.
      
      Unfortunately, FileStorage was one such mixed class, as was Connection.
      
      Moving to new-style classes seems to have a positive impact in the
      benchmarks. Here's zodbshootout on PyPy2 5.7.1 against a
      FileStorage (running in --threads and with 5 reps to be sure we get
      warmed up). First, current ZODB:
      
      "Transaction",                fs
      "Add 1000 Objects",            31,738
      "Update 1000 Objects",         42,444
      "Read 1000 Cold Objects",      51,894
      "Read 1000 Hot Objects",       53,187
      "Read 1000 Steamin' Objects", 835,877
      
      And with this PR:
      
      "Transaction",                fs
      "Add 1000 Objects",             35,651
      "Update 1000 Objects",          54,906
      "Read 1000 Cold Objects",      103,484
      "Read 1000 Hot Objects",        84,721
      "Read 1000 Steamin' Objects", 2,112,095
      
      The tests that hit the storage extensively are notably faster, as are
      steamin and hot, Connection having been a mixed class.
      
      I ran all tests multiple times. The data files were removed between
      runs. There's some variation, but the new-style classes always seem
      better.
      
      For comparison, here's CPython 2.7.13:
      
      "Transaction",                fs
      "Add 1000 Objects",            19,531
      "Update 1000 Objects",         16,201
      "Read 1000 Cold Objects",      22,111
      "Read 1000 Hot Objects",       21,851
      "Read 1000 Steamin' Objects", 880,582
      
      Locally I've run the tests under 2.7 and they all passed.
      e714f515
  16. 14 Apr, 2017 1 commit
    • Jason Madden's avatar
      Speed up ZODB.blob.BushyLayout.oid_to_path on Python 3 (#161) · 3580ddd8
      Jason Madden authored
      Profiling (https://github.com/zodb/zodbshootout/pull/32/) showed that
      this method was the only blob-related method that showed up in a test
      of creating blobs, other than those that actually performed IO.
      
      With this change its total and cumulative time drops from 0.003/0.004
      to 0.001/0.002 in a small benchmark. Blobs created per second shows a
      small but consistent improvement.
      
      Before:
      
         ncalls  tottime  percall  cumtime  percall filename:lineno(function)
            100    0.005    0.000    0.005    0.000 {built-in method rename}
            100    0.004    0.000    0.004    0.000 {function BlobFile.close at 0x1080d3a60}
            200    0.003    0.000    0.004    0.000 blob.py:576(oid_to_path)
            101    0.003    0.000    0.003    0.000 {built-in method mkdir}
            100    0.002    0.000    0.002    0.000 blob.py:333(__init__)
            402    0.002    0.000    0.005    0.000 {method 'dump' of '_pickle.Pickler' objects}
              1    0.002    0.002    0.034    0.034 Connection.py:553(_store_objects)
            201    0.002    0.000    0.002    0.000 {built-in method stat}
           5633    0.001    0.000    0.002    0.000 {built-in method isinstance}
      
      After:
      
         ncalls  tottime  percall  cumtime  percall filename:lineno(function)
            100    0.005    0.000    0.005    0.000 {built-in method rename}
            101    0.005    0.000    0.005    0.000 {built-in method mkdir}
            100    0.004    0.000    0.004    0.000 {function BlobFile.close at 0x10636aa60}
            402    0.002    0.000    0.005    0.000 {method 'dump' of '_pickle.Pickler' objects}
            100    0.002    0.000    0.002    0.000 blob.py:333(__init__)
              1    0.002    0.002    0.035    0.035 Connection.py:553(_store_objects)
            201    0.002    0.000    0.002    0.000 {built-in method stat}
           4033    0.001    0.000    0.001    0.000 {built-in method isinstance}
         ....
            200    0.001    0.000    0.002    0.000 blob.py:576(oid_to_path)
      3580ddd8
  17. 11 Apr, 2017 3 commits