Skip to content

GitLab

Switch branch/tag
  1. 24 Feb, 2012 1 commit
    • Julien Muchembled's avatar
      Implements backup using specialised storage nodes and relying on replication · 8e3c7b01
      Julien Muchembled authored 
      Replication is also fully reimplemented:
- It is not done anymore on whole partitions.
- It runs at lowest priority not to degrades performance for client nodes.

Schema of MySQL table is changed to optimize storage layout: rows are now
grouped by age, for good partial replication performance.
This certainly also speeds up simple loads/stores.
      8e3c7b01
  3. 21 Feb, 2012 1 commit
  5. 11 Oct, 2011 2 commits
    • Julien Muchembled's avatar
      Fix protocol and DB schema so that storages can handle transactions of any size · d5c469be
      Julien Muchembled authored 
      - Change protocol to use SHA1 for all checksums:
  - Use SHA1 instead of CRC32 for data checksums.
  - Use SHA1 instead of MD5 for replication.

- Change DatabaseManager API so that backends can store raw data separately from
  object metadata:
  - When processing AskStoreObject, call the backend to store the data
    immediately, instead of keeping it in RAM or in the temporary object table.
    Data is then referenced only by its checksum.
    Without such change, the storage could fail to store the transaction due to
    lack of RAM, or it could make tpc_finish step very slow.
  - Backends have to store data in a separate space, and remove entries as soon
    as they get unreferenced. So they must have an index of checksums in object
    metadata space. A new '_uncommitted_data' backend attribute keeps references
    of uncommitted data.
  - New methods: _pruneData, _storeData, storeData, unlockData
  - MySQL: change vertical partitioning of 'obj' by having data in a separate
    'data' table instead of using a shortened 'obj_short' table.
  - BTree: data is moved from '_obj' to a new '_data' btree.

- Undo is optimized so that backpointers are not required anymore to fetch data:
  - The checksum of an object is None only when creation is undone.
  - Removed DatabaseManager methods: _getObjectData, _getDataTIDFromData
  - DatabaseManager: move some code from _getDataTID to findUndoTID so that
    _getDataTID only has what's specific to backend.

- Removed because already covered by ZODB tests:
  - neo.tests.storage.testStorageDBTests.StorageDBTests.test__getDataTID
  - neo.tests.storage.testStorageDBTests.StorageDBTests.test__getDataTIDFromData
      d5c469be
    • Julien Muchembled's avatar
      Allow NEO to store empty values · d90c5b83
      Julien Muchembled authored 
      This changes how NEO stores undo information
and how it is transmitted on the network.
      d90c5b83
  7. 05 Sep, 2011 1 commit
  9. 27 Apr, 2011 2 commits
    • Julien Muchembled's avatar
      storage: teach shutdown to close all connections · 15308c68
      Julien Muchembled authored 
      git-svn-id: https://svn.erp5.org/repos/neo/trunk@2735 71dcc9de-d417-0410-9af5-da40c76e7ee4
      15308c68
    • Julien Muchembled's avatar
      connection: make close always call handler (connectionClosed or connectionFailed) · 4b6c1387
      Julien Muchembled authored 
      Main reason is that it's difficult to know in advance which side really closes
the connection. Network events can be chaotic and this could lead to many race
conditions.
Thus, handler can be used to update any database that is somewhat redundant
to the connection status, i.e. node status usually. Safely and less duplicated
code.

This change is motivated by recurrent random failures during election.
An example of race condition was that 2 fully connected master could close the
extra connection (the primary -> secondary one) at the same time.

In order to stabilize lower-level code and start with reliable election process,
code has also been simplified to not care about node states. All connections
without exception are closed at the end of the election and states are then
updated 1 by 1 by identification handler.
Note that during election, there may be 2 connection per node, which makes
difficult to update node states by connectionFailed/connectionClosed events.

timeoutExpired & peerBroken are dropped as they are unused for the moment.
A new API should be designed so that connectionClosed know the reason of the
close.
BROKEN state becomes unused.

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2732 71dcc9de-d417-0410-9af5-da40c76e7ee4
      4b6c1387
  11. 17 Jan, 2011 1 commit
  13. 06 Jan, 2011 1 commit
    • Vincent Pelletier's avatar
      Resolve deadlocks detected by storage nodes. · 7fccafeb
      Vincent Pelletier authored 
      Also reverts r2533: even if it's true that TIDs have no meaning at that level,
they are a handy way to prioritise a transaction over another, to break lock
cycles (aka deadlocks). This is the "detection" part of the change.
When a storage reports a deadlock, client must store all already-stored
objects again with "unlock" flag set.
Upon receiving those store requests, storage must release locks held by
transaction on those objects, and requeue the store request. If client didn't
hold any lock (initial store was still in queue), drop the second store
request.
This doesn't solve possible deadlocks if ZODB-level sends us objects in a
different order (ex: client 1 sending [OID1, OID2] & client 2 sending
[OID2, OID1]).
There is one important change to r2533's revert, which queues older
transactions and notifies deadlock for younger ones. The original code did it
the other way around, and it looks unfair to old transactions (they will keep
being delayed by younger ones, and will just get older and older).

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2596 71dcc9de-d417-0410-9af5-da40c76e7ee4
      7fccafeb
  15. 29 Oct, 2010 1 commit
  17. 28 Aug, 2010 1 commit
  19. 24 Aug, 2010 1 commit
    • Vincent Pelletier's avatar
      Improve replication SQL queries. · d8a7a177
      Vincent Pelletier authored 
      It is more efficient to provide a boundary value than a row count range.
This fixes replication on partitions with a large number of objects, revisions
or transactions: query time is now constant where it used to increase, causing
timeout problems when query duration exceeded ping time + ping timeout (11s
currently).

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2221 71dcc9de-d417-0410-9af5-da40c76e7ee4
      d8a7a177
  21. 21 Jun, 2010 1 commit
    • Grégory Wisniewski's avatar
      Move stored OIDs check to master side. · 1f629dfc
      Grégory Wisniewski authored 
      - The storages no more check the last OID during a store
- The storages inconditionnaly store the last OID notified by the master
- The master check during the if a greater oid was used by a client
- The master always notify the last OID when a pool is generated or if the
check above is True
- The master's transaction manager manager the last oid and oid generator

git-svn-id: https://svn.erp5.org/repos/neo/trunk@2180 71dcc9de-d417-0410-9af5-da40c76e7ee4
      1f629dfc
  23. 15 May, 2010 1 commit
  25. 26 Mar, 2010 1 commit
    • Vincent Pelletier's avatar
      Use value_serial for undo support. · 83c02447
      Vincent Pelletier authored 
      This mimics what FileStorage uses (file offsets) but in a relational manner.
This offloads decision of the ability to undo a transaction to storages,
avoiding 3 data loads for each object in the transaction at client side.
This also makes Neo refuse to undo transactions where object data would happen
to be equal between current value and undone value.
Finally, this is required to make database pack work properly (namely, it
prevents loosing objects which are orphans at pack TID, but are reachable
after it thanks to a transactional undo).

Also, extend storage's transaction manager so database adapter can fetch data
already sent by client in the same transaction, so it can undo multiple
transactions at once. Requires making object lock re-entrant (done in this
commit).

git-svn-id: https://svn.erp5.org/repos/neo/trunk@1978 71dcc9de-d417-0410-9af5-da40c76e7ee4
      83c02447
  27. 25 Feb, 2010 1 commit
  29. 23 Feb, 2010 1 commit
  31. 19 Feb, 2010 1 commit
  33. 16 Feb, 2010 1 commit
  35. 12 Feb, 2010 1 commit
  37. 05 Feb, 2010 1 commit
  39. 03 Feb, 2010 1 commit
  41. 02 Feb, 2010 1 commit
  43. 01 Feb, 2010 2 commits
  45. 28 Jan, 2010 1 commit
  47. 13 Jan, 2010 1 commit
  49. 02 Nov, 2009 1 commit
  51. 07 Oct, 2009 1 commit
  53. 05 Oct, 2009 1 commit
  55. 01 Oct, 2009 3 commits
  57. 30 Sep, 2009 1 commit
  59. 29 Sep, 2009 1 commit
  61. 10 Aug, 2009 1 commit
  63. 03 Aug, 2009 1 commit
  65. 31 Jul, 2009 1 commit
  67. 29 Jul, 2009 2 commits
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7