GitLab

Variables are investigated, recursively in case of container objects (like lists, for example),
to detect if they can be stored in the ZODB.
In this investigation persistent objects are identified by being an instance of the object
class and implementing a `__getstate__` method that raises no exception. If the variable is
not a Persistent object then we try to pickle and load it.

While developing the pickleable object identification a complication was found. It seems that
the code cannot capture cPickle.PicklingError in the usual way, `except cPickle.PicklingError`.
It's consequence of some weirdness with regards to pickle/cPickle modules exceptions classes and
more about it can be read at http://bugs.python.org/issue1457119. So, the workaround for this complication
was to catch all exceptions and check the exception class name as string.

The whole check for zodb persistence was moved into an utility function for the sake of readability
and code maintenance.

The Base_executeJupyter script object was transformed into an extension to be able to properly handle
transaction errors and render them correctly inside Jupyter.

erp5_data_notebook: moves variables and setup storage from ActiveProcess to `notebook_context` property

Notebook's variables and setup were being stored in an ActiveProcess, which is
not needed anymore, because now everything related to user context can be
safely stored in the ZODB.
They are stored in a `notebook_context` property of the Data Notebook
itself. Code and tests were updated properly. The old `process` property was removed.

All the references to *_variable_dict were renamed to  *_notebook_context, documentation
and tests were updated. Related objects like scripts and external methods were renamed
too.

To store objects in the `notebook_context` property we do 2 different tests: the first
to check if the object can be serialized by `ZODB.serialize.ObjectWriter`. If the first
test fails we test if the object can be serialized by cPickle. For the second test we
need to dump & load the object to be completely sure that it can be correctly loaded
later.

The function called by the Base_runJupyterCode external method was renamed from
Base_compileJupyterCode to Base_runJupyterCode be more consistent and avoid confusion.

All errors while running setup functions and now properly propagated to the user
interface in Jupyter and code execution is aborted.

- An environment object was implemented to help us deal with the multiprocess
architecture of ERP5 and objects that cannot be easily stored in the ZODB.
It stores definition of functions, classes and variables as string. The implementation
uses a dumb Environment class to allow users to make `define` and `undefine` calls,
which are captured and processed by an AST transformer before code execution.

- Along with the environment object, an automatic "import fixer" was created. It does
not allow users to import modules as they normally would, because this may cause
collateral effects on other users' code. A good example is the plot settings in the
matplotlib module. It will fix normal imports, make them use the environment object
mentione earlier automatically and warn the user about it.

A few bugs were fixed with this implementation, for example:

- https://nexedi.erp5.net/bug_module/20160318-7098DD, which reports an inconsistency
on portal catalog queries between Jupyter and Python (Script) objects. Probably an
issue with user context storage in ActiveProcess

- https://nexedi.erp5.net/bug_module/20160330-13AA193, which reports an error related
to acquisition when trying to plot images, which happened in other situations, although
this is not officially reported in Nexedi's ERP5. This probably also was happening because
of old user context storage.

This is the proper way to refer to the URL on which ERP5 is being served.

pivottablejs is a very useful pivot table implementation in Javascript that
alllows the user to create his own tables and charts. And also they had examples
of integration with Pandas.DataFrame objects and Jupyter. So this is highly
based on that.

**ATTENTION**: this is an experimental integration and does not follow the ERP5
Javascript standards. It will be refactored in the future to use RenderJS and
JIO.

The integration generates an HTML page template which starts the pivot table and
have a placeholder for the data, that will be later replaced with a Data Frame
data as CSV. After this replacement the page is stored in the memcached server
and then served from there, through a Script (Python) object, inside an HTML
iframe. The iframe is necessary because a lot of Javascript libraries that are
not included in the Jupyter web page are loaded.

A web page with id "PivotTableJs_getMovementHistoryList" was created to demo
how pivottablejs can be integrated within ERP5, either using AJAX or not.

In the process of this integration a simple external method to render
iPython's display classes (Images, Video, Youtube, IFrame, etc) was created. It
will be refactored and polished along with the kernel itself in the future.

@kirr, @Tyagov and @tatuya, please review.

Now the ERP5 Jupyter kernel automatically renders errors that happens in the
user-side code. Errors are captured during the AST tree creation (to be able to
detect syntax errors) and at execution time. The current transaction is
automatically aborted on error detection.

/reviewed-on nexedi/erp5!85

erp5_data_notebook: Refactor test because mime_type for an invalid image shouldn't be dependent on the filename only

Problem: Module objects are not picklable, thus, trying them to save in
local_variable_dict which is further saved in ZODB via ActiveResult object
of CMFActivity was giving error. So, its better to save the module objects
as code_strings in local_variable_dict.

As for now, for example:
local_variable_dict = {'imports': ['import numpy as np', 'import matplotlib as mpl'],
			 'variables': {'a':1, 'b:2'}}
for jupyter_code = """
import numpy as np
import matplotlib as mlp
a= 1; b=2; print a+b
"""

erp5_data_notebook bt5: Refactor test to use reference while searching through portal_catalog instead of notebook_code which was incorrect as there was no key as such in SQL_catalog for erp5

erp5_data_notebook bt5: Changed name of portal type from Data Notebook Message to Data Notebook Line and changes in various scripts where it has been used

erp5-data-notebook bt5: Refactor testExecuteJupyter to new JupyterCompile where we are letting errors fails loudly in erp5

This BT is used to interact ERP5 with Jupyter and execute the python code.
1. External method `Base_executeJupyter` to interact between code from Jupyter and ERP5 backend.
2. Module - Data Notebooks
3. Unittest `JupyterCompile` to test external method and extensions.