Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin.core
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
1
Issues
1
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
nexedi
wendelin.core
Commits
a47d1fbb
Commit
a47d1fbb
authored
Jun 27, 2016
by
Kirill Smelkov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
readme: Merge in wiki content
Upon
@jp
request.
parent
c687102c
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
46 additions
and
6 deletions
+46
-6
README.rst
README.rst
+46
-6
No files found.
README.rst
View file @
a47d1fbb
...
@@ -63,15 +63,55 @@ several nodes in a cluster.
...
@@ -63,15 +63,55 @@ several nodes in a cluster.
Please see `demo/demo_zbigarray.py`__ for a complete example.
Please see `demo/demo_zbigarray.py`__ for a complete example.
Additional Info
__ demo/demo_zbigarray.py
---------------
Current state and Roadmap
=========================
Wendelin.core works in real life for workloads Nexedi_ is using in production,
including 24/7 projects. We are, however, aware of the following
limitations and things that need to be improved:
- wendelin.core is currently not very fast
- there are big - proportional to input in size - temporary array allocations
in third-party libraries (NumPy_, `scikit-learn`_, ...) which might practically
prevent processing out-of-core arrays depending on the functionality used.
Thus
- we are currently working on improved wendelin.core design and implementation,
which will use kernel virtual memory manager (instead of one implemented__ in__
userspace__) with arrays backend presented to kernel via FUSE as virtual
filesystem implemented in Go.
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/include/wendelin/bigfile/virtmem.h
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/virtmem.c
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/pagefault.c
In parallel we will also:
- try wendelin.core 1.0 on large data sets
- identify and incrementally fix big-temporaries allocation issues in NumPy and
scikit-learn
We are open to community help with the above.
Additional materials
====================
- Wendelin.core tutorial__
- Slides__ (pdf__) from presentation about wendelin.core in PyData Paris 2015
__ https://www.nexedi.com/wendelin-Core.Tutorial.2016
__ http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret/asEntireHTML
__ http://www.wendelin.io/NXD-Wendelin.Core.Non.Secret?format=pdf
Please see `wiki`__ for more info: current state, roadmap, additional materials, ...
.. _NumPy: http://www.numpy.org/
.. _NumPy: http://www.numpy.org/
.. _scikit-learn: http://scikit-learn.org/
.. _numpy.memmap: http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html
.. _numpy.memmap: http://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html
.. _NEO: http://www.neoppod.org/
.. _NEO: http://www.neoppod.org/
.. _ZEO: https://pypi.python.org/pypi/ZEO
.. _ZEO: https://pypi.python.org/pypi/ZEO
.. _Nexedi: https://www.nexedi.com/
__ demo/demo_zbigarray.py
__ https://lab.nexedi.com/nexedi/wendelin.core/wikis/home
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment