Skip to content

J
jio
Commit 30f1edbb authored by lucas.parsy's avatar lucas.parsy
Browse files
Options

updated documentation 

renamed "revision_storage.rst" "replicate_storage.rst"
you need to "compile" the files (go in doc/ directory and type "make")
to have the html files.
parent 65abad13
Expand all Show whitespace changes
Inline Side-by-side
Showing with 472 additions and 527 deletions
docs/available_storages.rst
View file @ 30f1edbb
This diff is collapsed.
docs/developers.rst
View file @ 30f1edbb
......@@ -12,7 +12,7 @@ If you want to modify or build jIO yourself, you need to
* Clone from a repository
``$ git clone https://github.com/nexedi/jio.git``
``$ git clone https://lab.nexedi.com/nexedi/jio.git``
* Install `NodeJS <http://nodejs.org/>`_ (including ``npm``)
......@@ -53,8 +53,8 @@ Create a constructor:
}
}
Create 10 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachment``,
``remove``, ``removeAttachment``, ``allDocs``, ``check`` and ``repair``.
Create 9 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachment``,
``remove``, ``removeAttachment``, ``allDocs``. ``repair`` method is optional.
.. code-block:: javascript
......@@ -81,66 +81,26 @@ Create 10 methods: ``post``, ``put``, ``putAttachment``, ``get``, ``getAttachmen
(To help you design your methods, some tools are provided by jIO.util.)
The first parameter command provides some methods to act on the jIO job:
* ``success``, to tell jIO that the job is successfully terminated
``command.success(status[Text], [{custom key to add to the response}]);``
* ``resolve``, is equal to success
* ``error``, to tell jIO that the job cannot be done
``command.error(status[Text], [reason], [message], [{custom key to add to the response}])``
* ``retry``, to tell jIO that the job cannot be done now, but can be retried later. (same API than error)
* ``reject``, to tell jIO that the job cannot be done, let jIO to decide whether to retry or not. (same API than error)
The second parameter ``metadata`` or ``param`` is the first parameter provided by the jIO user.
The third parameter ``option`` is the option parameter provided by the jIO user.
Methods should return the following objects:
Methods should return:
* **post()** --> ``success("created", {"id": new_generated_id})``
* **post()**, **put()**, **remove()** --> id of the document affected (string)
* **put()**, **remove()**, **putAttachment()** or **removeAttachment()** --> ``success(204)``
* **putAttachment()** or **removeAttachment()** --> no specific value
* **get()** --> ``success("ok", {"data": document_metadata})``
* **get()** --> document_metadata (object)
* **getAttachment()** -->
.. code-block:: javascript
success("ok", {"data": binary_string, "content_type": content_type})
// or
success("ok", {"data": new Blob([data], {"type": content_type})})
* **allDocs()** --> ``success("ok", {"data": row_object})``
* **check()** -->
.. code-block:: javascript
// if metadata provides "_id" -> check document state
// if metadata doesn't promides "_id" -> check storage state
success("no_content")
// or
error("conflict", "corrupted", "incoherent document or storage")
* **repair()** -->
new Blob([data], {"type": content_type})
.. code-block:: javascript
* **allDocs()** --> list of all documents (restricted by a query, if given). (object)
// if metadata provides "_id" -> repair document state
// if metadata doesn't promides "_id" -> repair storage state
success("no_content")
// or
error("conflict", "corrupted", "impossible to repair document or storage")
// DON'T DESIGN STORAGES IF THERE IS NO WAY
// TO REPAIR INCOHERENT STATES
After creating all methods, your storage must be added to jIO. This is done
with the ``jIO.addStorage()`` method, which requires two parameters: the storage
......@@ -154,160 +114,3 @@ type (string) and a constructor (function). It is called like this:
Please refer to *localstorage.js* implementation for a good example on how to
setup a storage and what methods are required.
Also keep in mind that jIO is a job-based library: whenever you trigger a method,
a job is created, which will later return a response, after being processed.
Job rules
---------
The jIO job manager follows several rules set at the creation of a new jIO
instance. When you try to call a method, jIO will create a job and will make
sure the job is really necessary and will be executed. Thanks to these job
rules, jIO knows what to do with the new job before adding it to the queue. You
can also add your own rules, as we're going to see now.
These are the jIO **default rules**:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "readers update",
"conditions": [
"sameStorageDescription",
"areReaders",
"sameMethod",
"sameParameters",
"sameOptions"
],
"action": "update"
}, {
"code_name": "metadata writers update",
"conditions": [
"sameStorageDescription",
"areWriters",
"useMetadataOnly",
"sameMethod",
"haveDocumentIds",
"sameParameters"
],
"action": "update"
}, {
"code_name": "writers wait",
"conditions": [
"sameStorageDescription",
"areWriters",
"haveDocumentIds",
"sameDocumentId"
],
"action": "wait"
}]
});
The following actions can be used:
* ``ok`` - accept the job
* ``wait`` - wait until the end of the selected job
* ``update`` - bind the selected job to this one
* ``deny`` - reject the job
The following condition functions can be used:
* ``sameStorageDescription`` - check if the storage descriptions are different.
* ``areWriters`` - check if the commands are ``post``, ``put``, ``putAttachment``, ``remove``, ``removeAttachment``, or ``repair``.
* ``areReaders`` - check if the commands are ``get``, ``getAttachment``, ``allDocs`` or ``check``.
* ``useMetadataOnly`` - check if the commands are ``post``, ``put``, ``get``, ``remove`` or ``allDocs``.
* ``sameMethod`` - check if the commands are equal.
* ``sameDocumentId`` - check if the document ids are equal.
* ``sameParameters`` - check if the metadata or param are equal (deep comparison).
* ``sameOptions`` - check if the command options are equal.
* ``haveDocumentIds`` - test if the two commands contain document ids.
Create Job Condition
--------------------
You can create two types of function: job condition, and job comparison.
.. code-block:: javascript
// Job Condition
// Check if the job is a get command
jIO.addJobRuleCondition("isGetMethod", function (job) {
return job.method === 'get';
});
// Job Comparison
// Check if the jobs have the same 'title' property
// only if they are strings
jIO.addJobRuleCondition("sameTitleIfString",
function (job, selected_job) {
if (typeof job.kwargs.title === 'string' &&
typeof selected_job.kwargs.title === 'string') {
return job.kwargs.title === selected_job.kwargs.title;
}
return false;
});
Add job rules
-------------
You just have to define job rules in the jIO options:
.. code-block:: javascript
// Do not accept to post or put a document which title is equal
// to another already running post or put document title
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "avoid similar title",
"conditions": [
"sameStorageDescription",
"areWriters",
"sameTitleIfString"
],
"action": "deny",
"before": "writers update" // optional
// "after": also exists
}]
});
Clear/Replace default job rules
-------------------------------
If a job's ``code_name`` is equal to ``readers update``, then adding this rule will replace it:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"job_rules": [{
"code_name": "readers update",
"conditions": [
"sameStorageDescription",
"areReaders",
"sameMethod",
"haveDocumentIds"
"sameParameters"
// sameOptions is removed
],
"action": "update"
}]
});
Or you can just clear all rules before adding new ones:
.. code-block:: javascript
var jio_instance = jIO.createJIO(storage_description, {
"clear_job_rules": true,
"job_rules": [{
// ...
}]
});
docs/getting_started.rst
View file @ 30f1edbb
......@@ -79,8 +79,6 @@ Getting started
| Deletes a document's attachment
``.allDocs()`` | ``my_jio.allDocs([options]);``
| Retrieves a list of existing documents
``.check()`` | ``my_jio.check(document, [options]);``
| Checks the document state
``.repair()`` | ``my_jio.repair(document, [options]);``
| Repairs the document
======================= ======================================================
......@@ -122,29 +120,29 @@ Storage dependencies
Storage connectors
^^^^^^^^^^^^^^^^^^
* localstorage.js
* davstorage.js
* searchableencryptionstorage.js (depends on sjcl) (WIP)
* s3storage.js (depends on sha1, jQuery) (WIP)
* Localstorage
* MemoryStorage
* IndexedDB
* WebSQL
* DavStorage
* Dropbox
* Google Drive
* ERP5Storage
* s3storage.js (WIP)
* xwikistorage.js (depends on jQuery) (WIP)
* erp5storage.js (WIP)
* restsqlstorage.js (depends on jQuery) (WIP)
* mioga2storage.js (depends on jQuery) (WIP)
Storage handlers
^^^^^^^^^^^^^^^^
* indexstorage.js
* gidstorage.js
* splitstorage.js (WIP)
* replicatestorage.js (WIP)
Revision based storage handlers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* revisionstorage.js (depends on sha256)
* replicaterevisionstorage.js
* Zipstorage
* ShaStorage
* UUIDStorage
* QueryStorage
* CryptStorage
* UnionStorage
* FileSystemBridgeStorage
* Document Storage
* Replicate Storage
Unit tests
^^^^^^^^^^
......@@ -155,9 +153,5 @@ the test suite with each release.
Fork jIO
^^^^^^^^
The same source code is kept in three synchronized repositories.
Feel free to use any of them.
* `GitHub <https://github.com/nexedi/jio>`_: ``git clone https://github.com/nexedi/jio.git``
* `Gitorius <https://gitorious.org/nexedi/jio>`_: ``git clone https://git.gitorious.org/nexedi/jio.git``
* `Git Erp5 <http://git.erp5.org/gitweb/jio.git>`_ (read only): ``git clone http://git.erp5.org/repos/jio.git``
| Feel free to use the Gitlab repository:
| `GitLab <https://lab.nexedi.com/nexedi/jio.git>`_: ``git clone https://lab.nexedi.com/nexedi/jio.git``
docs/index.rst
View file @ 30f1edbb
......@@ -38,9 +38,8 @@ jIO documentation
getting_started
manage_documents
revision_storages
replicate_storage
available_storages
gid_storage
query
keys
metadata
......
docs/manage_documents.rst
View file @ 30f1edbb
This diff is collapsed.
docs/metadata.rst
View file @ 30f1edbb
......@@ -58,11 +58,6 @@ List of metadata to use
Identification
^^^^^^^^^^^^^^
* **_id**
A specific jIO metadata which helps the storage to find a document
(can be a real path name, a dc:identifier, a uuid, ...). **String Only**
* **identifier**
| ``{"identifier": "http://domain/jio_home_page"}``
......
docs/replicate_storage.rst 0 → 100644
View file @ 30f1edbb
.. _replicate-storage-conflicts-and-resolution:
Replicate Storage: Conflicts and Resolution
===========================================
Why Conflicts can Occur
-----------------------
Using jIO you can store documents in multiple locations. With an
increasing number of users working on a document and some storages not being
available or responding too slow, conflicts are more likely to occur. jIO
defines a conflict as multiple versions of a document existing in a storage
tree and a user trying to save on a version that does not match the latest
version of the document.
To keep track of document versions a replicate storage must be used. When doing
so, jIO creates a document tree file for every document. This file contains all
existing versions and their status and is modified whenever a version is
added/updated/removed or when storages are being synchronized.
How conflicts are handled
-------------------------
The RemoteStorage takes in parameter two substorages, one "local" and one "remote".
The "local" storage can be remote, but it will be used for all the requests
like **get()**, **getAttachment()**, **allDocs()**...
Using the document tree, jIO tries to make every version of a document
available on the two storages. When multiple versions of a document exist,
Jio will follow the rule set by the conflict_handling option, given at storage creation.
This option can one of the following numbers:
* 0: no conflict resolution (throws an error when conflict is occuring)
* 1: keep the local state. (overwrites the remote document with local content)
* 2: keep the remote state. (overwrites the local document with remote content)
* 3: keep both copies (leave documents untouched, no signature update)
Simple Conflict Example
-----------------------
.. TODO this is a little confusing
You are keeping a namecard file on your PC updating from your smartphone. Your
smartphone ran out of battery and is offline when you update your namecard on
your PC with your new email adress. Someone else changes this email from your PC
and once your smartphone is recharged, you go back online and the previous
update is executed.
#. Set up the replicate storage:
.. code-block:: javascript
var jio_instance = jIO.createJIO({
// replicate storage
type: 'replicate',
local_sub_storage : {
type: 'local',
...
}
remote_sub_storage: {
type: 'dav',
...
}
conflict_handling: ...
});
#. 1) Create the namecard on your smartphone:
.. code-block:: javascript
jio_instance.put("myNameCard", {
email: 'jb@td.com'
}).then(function (response) {
// response -> 'myNameCard'
});
This will create the document on your WebDAV and local storage
#. 2) Someone else updates your shared namecard on WebDAV:
.. code-block:: javascript
jio_instance.put(myNameCard, {
email: 'kyle@td.com',
}).then(function (response) {
// response -> 'myNameCard'
});
Your smartphone is offline, so now you will have one version on
your smartphone and another version on WebDAV on your PC.
#. 3) Later, your smartphone is online and you modify your email:
.. code-block:: javascript
jio_instance.get('myNameCard').then(function (response) {
// response.email -> 'jb@td.com'
// the get() method checks only on your local storage
// and doesn't warn you about remote modifications.
return jio_instance.put('myNameCard', {
email: 'jack@td.com'
})
.then(function (response) {
// response -> 'myNameCard'
});
| Your latest modification of the email is: "jack@td.com"
| The modification from the other user is: "kyle@td.com"
If your conflict_handling option was:
* | 0: the email is:
| -"kyle@td.com" on WebDAV
| -"jack@td.com" on your local storage
| The storage rejects your latest modification,
| you get an error because local and remote documents are desynchronized.
| The documents in local and remote state are left untouched.
* | 1: the email is: "jack@td.com" on both storages
| The storage pushes the local modification, which is yours.
* | 2: the email is: "kyle@td.com" on both storages
| The storage keeps the remote modification, which is from the other user.
| Your local storage is modified to fit the state of the remote storage.
* | 3: the email is: "jack@td.com" on both storages
| The storage doesn't do synchronization, and pushes your modification
| without checking if the remote storage has been changed or not
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7