Skip to content

erp5
erp5
Commit d67f9a7f authored by Douglas's avatar Douglas
Browse files
Options

Jupyter: automatic rendering of objects as HTML 

Rendering of last-returning objects wasn't implemented using a custom display
hook, like in IPyKernel, because it is not thread-safe. Instead user's code is
modified to call an external method, Base_renderAsHtml, with the object as
argument to render it. This external method can be called manually as the user
wishes.

To render a object of a given class a processor needs to be created and
associated with that class. When the class has no processor, the default one is
used to render the object's string representation. Many processors were created
for some special classes: Image class from the Image Module, Figure class from
matplotlib and all the IPython's display objects classes (Audio, IFrame,
YouTubeVideo, VimeoVideo, ScribdDocument, FileLink, and FileLinks).

The user can manually add processors as he likes using the `_processor_list`
object that is automatically injected into the code context.
parent c32ff174
Expand all Show whitespace changes
Inline Side-by-side
Showing with 369 additions and 306 deletions
bt5/erp5_data_notebook/ExtensionTemplateItem/portal_components/extension.erp5.JupyterCompile.xml
View file @ d67f9a7f
...@@ -46,12 +46,14 @@ ...@@ -46,12 +46,14 @@
<key> <string>text_content_warning_message</string> </key> <key> <string>text_content_warning_message</string> </key>
<value> <value>
<tuple> <tuple>
<string>W: 55, 2: Using the global statement (global-statement)</string> <string>W: 85, 2: Redefining name \'traceback\' from outer scope (line 9) (redefined-outer-name)</string>
<string>W: 95, 8: Use of exec (exec-used)</string> <string>W:197, 8: Use of exec (exec-used)</string>
<string>W:119, 8: Use of exec (exec-used)</string> <string>W:252, 8: Use of exec (exec-used)</string>
<string>W:125, 8: Use of exec (exec-used)</string> <string>W:264, 8: Use of exec (exec-used)</string>
<string>W:220, 4: Using the global statement (global-statement)</string> <string>W:327, 10: Unused variable \'mime_type\' (unused-variable)</string>
<string>W:320, 2: Using the global statement (global-statement)</string> <string>W:462, 2: Redefining name \'IFrame\' from outer scope (line 17) (redefined-outer-name)</string>
<string>W: 9, 0: Unused import traceback (unused-import)</string>
<string>W: 13, 0: Unused import transaction (unused-import)</string>
</tuple> </tuple>
</value> </value>
</item> </item>
......
bt5/erp5_data_notebook/PathTemplateItem/web_page_module/PivotTableJs_getMovementHistoryList.html 0 → 100644
View file @ d67f9a7f
<html>
<head>
<script type="text/javascript" src="http://localhost:2200/erp5/jquery/core/jquery.min.js"></script>
<script type="text/javascript" src="http://localhost:2200/erp5/jquery/ui/js/jquery-ui.min.js"></script>
<script type="text/javascript" src="http://evanplaice.github.io/jquery-csv/src/jquery.csv.js"></script>
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.css">
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.js"></script>
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.css">
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/d3_renderers.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/c3_renderers.min.js"></script>
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/export_renderers.min.js"></script>
<script type="text/javascript">
$(document).ready(function () {
$('#filter_button').on('click', function (){
var data = $('form').serializeArray();
$('#pivottablejs').html('Loading...');
$.ajax({
type: "POST",
url: "http://localhost:2200/erp5/portal_skins/erp5_inventory_pandas/filterDataFrame?as_csv=True",
data: data,
success: function (response) {
var input = $.csv.toArrays(response);
$('#pivottablejs').pivotUI(input, {
renderers: $.extend(
$.pivotUtilities.renderers,
$.pivotUtilities.c3_renderers,
$.pivotUtilities.d3_renderers,
$.pivotUtilities.export_renderers
),
hiddenAttributes: [""],
rows: 'Sequence',
cols: 'Data'
});
},
error: function (response) {
$('#pivottablejs').html('Error while requesting data from server.');
}
})
});
});
</script>
</head>
<body>
<h1>Integration between Pandas-based Inventory API and PivotTableJS</h1>
<p><b>NOTE: for this protoype the code will use the Big Array object with title "Wendelin + Jupyter"</b></p>
<form>
<fieldset>
<legend>Is accountable?</legend>
<input type="radio" name="is_accountable" value="1" checked> Yes
<input type="radio" name="is_accountable" value="0"> No
</fieldset>
<fieldset>
<legend>Omit</legend>
<input type="checkbox" name="omit_input"> Omit Input
<input type="checkbox" name="omit_output"> Omit Output
<input type="checkbox" name="omit_asset_increase"> Omit Asset Increase
<input type="checkbox" name="omit_asset_decrease"> Omit Asset Decrease
</fieldset>
<fieldset>
<legend>Simulation State</legend>
<p>Simulation State: <input name="simulation_state"></p>
<p>Input Simulation State: <input name="input_simulation_state"></p>
<p>Output Simulation State: <input name="output_simulation_state"></p>
</fieldset>
<fieldset>
<legend>Dates</legend>
<p>From date (yyyy-mm-dd): <input name="from_date"></p>
<p>To date (yyyy-mm-dd): <input name="to_date"></p>
</fieldset>
<button type="button" id="filter_button">Filter!</button>
</form>
<div id="pivottablejs">
</div>
</body>
</html>
\ No newline at end of file
bt5/erp5_data_notebook/PathTemplateItem/web_page_module/PivotTableJs_getMovementHistoryList.xml
View file @ d67f9a7f
...@@ -105,101 +105,6 @@ ...@@ -105,101 +105,6 @@
<key> <string>short_title</string> </key> <key> <string>short_title</string> </key>
<value> <string>getMovementHistoryList</string> </value> <value> <string>getMovementHistoryList</string> </value>
</item> </item>
<item>
<key> <string>text_content</string> </key>
<value> <string encoding="cdata"><![CDATA[
<html>\n
<head>\n
<script type="text/javascript" src="http://localhost:2200/erp5/jquery/core/jquery.min.js"></script>\n
<script type="text/javascript" src="http://localhost:2200/erp5/jquery/ui/js/jquery-ui.min.js"></script>\n
<script type="text/javascript" src="http://evanplaice.github.io/jquery-csv/src/jquery.csv.js"></script>\n
\n
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.css">\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js"></script>\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/c3/0.4.10/c3.min.js"></script>\n
\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.js"></script>\n
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/pivot.min.css">\n
\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/d3_renderers.min.js"></script>\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/c3_renderers.min.js"></script>\n
<script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/pivottable/2.0.2/export_renderers.min.js"></script>\n
\n
<script type="text/javascript">\n
$(document).ready(function () {\n
$(\'#filter_button\').on(\'click\', function (){\n
var data = $(\'form\').serializeArray();\n
$(\'#pivottablejs\').html(\'Loading...\');\n
\n
$.ajax({\n
type: "POST",\n
url: "http://localhost:2200/erp5/portal_skins/erp5_inventory_pandas/filterDataFrame?as_csv=True",\n
data: data,\n
success: function (response) {\n
var input = $.csv.toArrays(response);\n
$(\'#pivottablejs\').pivotUI(input, {\n
renderers: $.extend(\n
$.pivotUtilities.renderers, \n
$.pivotUtilities.c3_renderers, \n
$.pivotUtilities.d3_renderers,\n
$.pivotUtilities.export_renderers\n
),\n
hiddenAttributes: [""],\n
rows: \'Sequence\',\n
cols: \'Data\'\n
});\n
},\n
error: function (response) {\n
$(\'#pivottablejs\').html(\'Error while requesting data from server.\');\n
}\n
})\n
});\n
});\n
</script>\n
</head>\n
<body>\n
<h1>Integration between Pandas-based Inventory API and PivotTableJS</h1>\n
<p><b>NOTE: for this protoype the code will use the Big Array object with title "Wendelin + Jupyter"</b></p>\n
\n
<form>\n
<fieldset>\n
<legend>Is accountable?</legend>\n
<input type="radio" name="is_accountable" value="1" checked> Yes\n
<input type="radio" name="is_accountable" value="0"> No\n
</fieldset>\n
\n
<fieldset>\n
<legend>Omit</legend>\n
<input type="checkbox" name="omit_input"> Omit Input\n
<input type="checkbox" name="omit_output"> Omit Output\n
<input type="checkbox" name="omit_asset_increase"> Omit Asset Increase\n
<input type="checkbox" name="omit_asset_decrease"> Omit Asset Decrease\n
</fieldset>\n
\n
<fieldset>\n
<legend>Simulation State</legend>\n
<p>Simulation State: <input name="simulation_state"></p>\n
<p>Input Simulation State: <input name="input_simulation_state"></p>\n
<p>Output Simulation State: <input name="output_simulation_state"></p>\n
</fieldset>\n
\n
<fieldset>\n
<legend>Dates</legend>\n
<p>From date (yyyy-mm-dd): <input name="from_date"></p>\n
<p>To date (yyyy-mm-dd): <input name="to_date"></p>\n
</fieldset>\n
\n
<button type="button" id="filter_button">Filter!</button>\n
</form>\n
\n
<div id="pivottablejs">\n
</div>\n
</body>\n
</html>
]]></string> </value>
</item>
<item> <item>
<key> <string>title</string> </key> <key> <string>title</string> </key>
<value> <string>Web Interface for getMovementHistoryList</string> </value> <value> <string>Web Interface for getMovementHistoryList</string> </value>
......
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_displayPivotTableFrame.py 0 → 100644
View file @ d67f9a7f
memcached_tool = context.getPortalObject().portal_memcached
memcached_dict = memcached_tool.getMemcachedDict(key_prefix='pivottablejs', plugin_path='portal_memcached/default_memcached_plugin')
return memcached_dict[key]
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_displayPivotTableFrame.xml
View file @ d67f9a7f
...@@ -48,13 +48,6 @@ ...@@ -48,13 +48,6 @@
</object> </object>
</value> </value>
</item> </item>
<item>
<key> <string>_body</string> </key>
<value> <string>memcached_tool = context.getPortalObject().portal_memcached\n
memcached_dict = memcached_tool.getMemcachedDict(key_prefix=\'pivottablejs\', plugin_path=\'portal_memcached/default_memcached_plugin\')\n
return memcached_dict[key]\n
</string> </value>
</item>
<item> <item>
<key> <string>_params</string> </key> <key> <string>_params</string> </key>
<value> <string>key</string> </value> <value> <string>key</string> </value>
......
bt5/erp5_data_notebook/SkinTemplateItem/portal_skins/erp5_data_notebook/Base_renderAsHtml.xml 0 → 100644
View file @ d67f9a7f
<?xml version="1.0"?>
<ZopeData>
<record id="1" aka="AAAAAAAAAAE=">
<pickle>
<global name="ExternalMethod" module="Products.ExternalMethod.ExternalMethod"/>
</pickle>
<pickle>
<dictionary>
<item>
<key> <string>_function</string> </key>
<value> <string>renderAsHtml</string> </value>
</item>
<item>
<key> <string>_module</string> </key>
<value> <string>JupyterCompile</string> </value>
</item>
<item>
<key> <string>id</string> </key>
<value> <string>Base_renderAsHtml</string> </value>
</item>
<item>
<key> <string>title</string> </key>
<value> <string></string> </value>
</item>
</dictionary>
</pickle>
</record>
</ZopeData>
bt5/erp5_data_notebook/TestTemplateItem/portal_components/test.erp5.testExecuteJupyter.py
View file @ d67f9a7f
...@@ -32,8 +32,10 @@ from Products.ERP5Type.tests.utils import createZODBPythonScript, removeZODBPyth ...@@ -32,8 +32,10 @@ from Products.ERP5Type.tests.utils import createZODBPythonScript, removeZODBPyth
import time import time
import json import json
import base64 import base64
import transaction import transaction
class TestExecuteJupyter(ERP5TypeTestCase): class TestExecuteJupyter(ERP5TypeTestCase):
def afterSetUp(self): def afterSetUp(self):
...@@ -73,7 +75,9 @@ class TestExecuteJupyter(ERP5TypeTestCase): ...@@ -73,7 +75,9 @@ class TestExecuteJupyter(ERP5TypeTestCase):
def testJupyterCompileErrorRaise(self): def testJupyterCompileErrorRaise(self):
""" """
Test if JupyterCompile portal_component raises error on the server side. Test if JupyterCompile portal_component correctly catches exceptions as
expected by the Jupyter frontend as also automatically abort the current
transaction.
Take the case in which one line in a statement is valid and another is not. Take the case in which one line in a statement is valid and another is not.
""" """
portal = self.getPortalObject() portal = self.getPortalObject()
...@@ -114,6 +118,7 @@ portal.%s() ...@@ -114,6 +118,7 @@ portal.%s()
transaction.abort() transaction.abort()
# Clear the portal cache from previous transaction # Clear the portal cache from previous transaction
self.portal.portal_caches.clearAllCache() self.portal.portal_caches.clearAllCache()
# Remove the ZODB python script created above # Remove the ZODB python script created above
removeZODBPythonScript(script_container, script_id) removeZODBPythonScript(script_container, script_id)
...@@ -360,19 +365,20 @@ import sys ...@@ -360,19 +365,20 @@ import sys
self.assertEquals(json.loads(result)['code_result'].rstrip(), 'imghdr') self.assertEquals(json.loads(result)['code_result'].rstrip(), 'imghdr')
self.assertEquals(json.loads(result)['mime_type'].rstrip(), 'text/plain') self.assertEquals(json.loads(result)['mime_type'].rstrip(), 'text/plain')
def testBaseDisplayImageERP5Image(self): def testERP5ImageProcessor(self):
""" """
Test the fucntioning of Base_displayImage external method of erp5_data_notebook Test the fucntioning of the ERP5ImageProcessor and the custom system
BT5 for ERP5 image object as parameter and change display hook too.
""" """
self.image_module = self.portal.getDefaultModule('Image') self.image_module = self.portal.getDefaultModule('Image')
self.assertTrue(self.image_module is not None) self.assertTrue(self.image_module is not None)
# Create a new ERP5 image object # Create a new ERP5 image object
reference = 'testBase_displayImageReference' reference = 'testBase_displayImageReference5'
data_template = '<img src="data:application/unknown;base64,%s" />'
data = 'qwertyuiopasdfghjklzxcvbnm<somerandomcharacterstosaveasimagedata>' data = 'qwertyuiopasdfghjklzxcvbnm<somerandomcharacterstosaveasimagedata>'
self.image_module.newContent( self.image_module.newContent(
portal_type='Image', portal_type='Image',
id='testBase_displayImageID', id='testBase_displayImageID5',
reference=reference, reference=reference,
data=data, data=data,
filename='test.png' filename='test.png'
...@@ -382,7 +388,7 @@ import sys ...@@ -382,7 +388,7 @@ import sys
# Call Base_displayImage from inside of Base_runJupyter # Call Base_displayImage from inside of Base_runJupyter
jupyter_code = """ jupyter_code = """
image = context.portal_catalog.getResultValue(portal_type='Image',reference='%s') image = context.portal_catalog.getResultValue(portal_type='Image',reference='%s')
context.Base_displayImage(image_object=image) image
"""%reference """%reference
local_variable_dict = {'imports' : {}, 'variables' : {}} local_variable_dict = {'imports' : {}, 'variables' : {}}
...@@ -391,7 +397,7 @@ context.Base_displayImage(image_object=image) ...@@ -391,7 +397,7 @@ context.Base_displayImage(image_object=image)
old_local_variable_dict=local_variable_dict old_local_variable_dict=local_variable_dict
) )
self.assertEquals(result['result_string'].rstrip(), base64.b64encode(data)) self.assertEquals(result['result_string'].rstrip(), data_template % base64.b64encode(data))
# Mime_type shouldn't be image/png just because of filename, instead it is # Mime_type shouldn't be image/png just because of filename, instead it is
# dependent on file and file data # dependent on file and file data
self.assertNotEqual(result['mime_type'], 'image/png') self.assertNotEqual(result['mime_type'], 'image/png')
...@@ -430,3 +436,33 @@ context.Base_displayImage(image_object=image) ...@@ -430,3 +436,33 @@ context.Base_displayImage(image_object=image)
python_expression=jupyter_code2 python_expression=jupyter_code2
) )
self.assertEquals(json.loads(result)['code_result'].rstrip(), 'sys') self.assertEquals(json.loads(result)['code_result'].rstrip(), 'sys')
def testPivotTableJsIntegration(self):
'''
This test ensures the PivotTableJs user interface is correctly integrated
into our Jupyter kernel.
'''
portal = self.portal
self.login('dev_user')
jupyter_code = '''
class DataFrameMock(object):
def to_csv(self):
return "column1, column2; 1, 2;"
my_df = DataFrameMock()
iframe = context.Base_erp5PivotTableUI(my_df, 'https://localhost:2202/erp5')
context.Base_renderAsHtml(iframe)
'''
reference = 'Test.Notebook.PivotTableJsIntegration %s' % time.time()
notebook = self._newNotebook(reference=reference)
result = portal.Base_executeJupyter(
reference=reference,
python_expression=jupyter_code
)
json_result = json.loads(result)
# The big hash in this string was previous calculated using the expect hash
# of the pivot table page's html.
pivottable_frame_display_path = 'Base_displayPivotTableFrame?key=853524757258b19805d13beb8c6bd284a7af4a974a96a3e5a4847885df069a74d3c8c1843f2bcc4d4bb3c7089194b57c90c14fe8dd0c776d84ce0868e19ac411'
self.assertTrue(pivottable_frame_display_path in json_result['code_result'])
bt5/erp5_data_notebook/TestTemplateItem/portal_components/test.erp5.testExecuteJupyter.xml
View file @ d67f9a7f
...@@ -43,7 +43,9 @@ ...@@ -43,7 +43,9 @@
<item> <item>
<key> <string>text_content_warning_message</string> </key> <key> <string>text_content_warning_message</string> </key>
<value> <value>
<tuple/> <tuple>
<string>W:457, 4: Unused variable \'notebook\' (unused-variable)</string>
</tuple>
</value> </value>
</item> </item>
<item> <item>
......
bt5/erp5_data_notebook/bt/description
View file @ d67f9a7f
...@@ -2,4 +2,5 @@ Interaction between Jupyter(IPython Notebook) and ERP5. ...@@ -2,4 +2,5 @@ Interaction between Jupyter(IPython Notebook) and ERP5.
!WARNING! !WARNING!
This business template is unsafe to install on a public server as one of the extensions uses eval and allows remote code execution. Proper security should be taken into account. This business template is unsafe to install on a public server as one of the extensions uses eval and allows remote code execution. Proper security should be taken into account.
This template includes a highly exprimental integration with PivotTableJs which doesn't follow ERP5 Javascript standards and will be refactored to use JIO and RenderJS.
!WARNING! !WARNING!
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7