1. 17 Jan, 2016 1 commit
    • Kirill Smelkov's avatar
      helloweb/ruby: Abandon usage of slapos.cookbook:wrapper · 22124832
      Kirill Smelkov authored
      It turned out we cannot currently use slapos.cookbook in software part
      of SR - the reason is slapos.cookbook egg depends on lxml egg, which in
      turn needs libxml to be also installed via slapos.
      
      For this reason stack/slapos.cfg has
      
          [slapos-cookbook]   # NOTE _not_ slapos.cookbook
          recipe = zc.recipe.egg
          eggs =
            ${lxml-python:egg}
          ...
      
      and lxml-python is lxml building recipe to build it together with
      libxml:
      
          [lxml-python]
          recipe = zc.recipe.egg:custom
          egg = lxml
      
          rpath =
            ${libxml2:location}/lib/
            ${libxslt:location}/lib/
            ${zlib:location}/lib/
          environment = lxml-python-env
          ...
      
      So underlying idea, as I understand it, is: every SR contains
      slapos-cookbook in parts and this way lxml-python gets build. Then when
      there is slapos.cookbook egg usage, it is already correctly built.
      
      BUT
      
      This works when such slapos.cookbook egg usage happens _only_ in instance
      part of an SR: otherwise, if buildout sees slapos.cookbook egg usage in
      some recipe, e.g. like it currently is in helloweb-ruby:
      
          [helloweb-ruby]
          recipe  = slapos.cookbook:wrapper
          ...
      
      it _first_ tries to install slapos.cookbook egg directly - as needed for
      recipes eggs are installed as a first step, _before_ further buildout
      processing. What happens then is that slapos.cookbook (note not "-") egg
      sources and dependencies are downloaded from pypi, including lxml egg,
      all are tried to build, and in lxml egg it fails this way:
      
          ...
          Processing lxml-3.5.0
          Writing /tmp/tmpLGK4xWbuild/lxml-3.5.0/setup.cfg
          Running setup.py -q bdist_egg --dist-dir /tmp/tmpLGK4xWbuild/lxml-3.5.0/egg-dist-tmp-DJvofa
          Building lxml version 3.5.0.
          Building without Cython.
          ERROR: /bin/sh: 1: xslt-config: not found
      
          ** make sure the development packages of libxml2 and libxslt are installed **
      
          Using build configuration of libxslt
          In file included from src/lxml/lxml.etree.c:323:0:
          src/lxml/includes/etree_defs.h:14:31: fatal error: libxml/xmlversion.h: No such file or directory
           #include "libxml/xmlversion.h"
                                         ^
          compilation terminated.
          Compile failed: command 'gcc' failed with exit status 1
          /tmp/tmpLGK4xWbuild/lxml-3.5.0/temp/xmlXPathInitGlEAOF.c:1:26: fatal error: libxml/xpath.h: No such file or directory
           #include "libxml/xpath.h"
                                    ^
          compilation terminated.
          *********************************************************************************
          Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
          *********************************************************************************
          error: Setup script exited with error: command 'gcc' failed with exit status 1
          An error occurred when trying to install lxml 3.5.0. Look above this message for any errors that were output by easy_install.
          Could't load zc.buildout entry point wrapper
          from slapos.cookbook:
          Couldn't install: lxml 3.5.0.
          While:
            Installing.
            Getting section helloweb-ruby.
            Initializing section helloweb-ruby.
            Installing recipe slapos.cookbook.
          Error: Couldn't install: lxml 3.5.0
      
      Previously it probably used to work because we had system libxml
      installed, and this way lxml compilation succeeded (but was incorrect
      from slapos point of view).
      
      ( The problem turned out to be already known somehow - see e.g. c7d00913
        "Initial neoppod commit." and look for "Note on LXML/END LXML" there )
      
      Solution could be: either fix slapos.cookbook installation via e.g.
      teaching buildout to take into account pre-dependencies for eggs (for
      lxml) or just to avoid using slapos.cookbook:wrapper for executable
      generation.
      
      While @kazuhiko is working on the first more-generic solution, here goes
      a simpler one to just make helloweb component alive again: like it is
      done in a lot of places (e.g. in software/kvm/) let's use
      collective.recipe.template to generate a short shell script.
      
      NOTE
      
      previously the command line was
      
          ${bundler:bundle} exec sh -c 'helloweb.rb "$@"' ${:_buildout_section_name_}
      
      but now it is the same with "$@" appended:
      
          exec ${bundler:bundle} exec sh -c 'helloweb.rb "$@"' ${:_buildout_section_name_} "$@"
      
      The reason is slapos.cookbook:wrapper uses slapos.recipe.librecipe.execute.generic_exec()
      in generated script, which appends sys.argv[1:] to the command-line implicitly:
      
          def generic_exec(args):
            ...
            os.execve(exec_list[0], exec_list + sys.argv[1:], exec_env)
                                              ^^^^^^^^^^^^^^
      
          https://lab.nexedi.com/nexedi/slapos/blob/54bbe0a9/slapos/recipe/librecipe/execute.py#L84
      
      that's why last "$@" was not present in original version.
      
      P.S.
      
      Otherwise currently slapos.cookbook is used only in instance parts of
      recipes in whole slapos.git
      
      /reviewed-by TrustMe
      /debugged-with @kazuhiko
      /cc @vpelletier
      22124832
  2. 15 Jan, 2016 1 commit
    • Aurel's avatar
      python-magic : do not fail to import the lib · 54bbe0a9
      Aurel authored
      ctypes lib does not read the rpath of the current binary to look after the
      desired library, so although it exits the import failed. Pass-by the first
      find to make the import work
      54bbe0a9
  3. 14 Jan, 2016 2 commits
  4. 13 Jan, 2016 1 commit
  5. 12 Jan, 2016 5 commits
  6. 08 Jan, 2016 1 commit
  7. 04 Jan, 2016 1 commit
  8. 28 Dec, 2015 4 commits
  9. 26 Dec, 2015 1 commit
  10. 23 Dec, 2015 1 commit
    • Julien Muchembled's avatar
      re6stnet: update and cleanup · 284612ac
      Julien Muchembled authored
      ../../stack/slapos.cfg is removed from component/*/buildout.cfg
      because we normally don't specify it in component/
      The OBS package will need to extend it.
      284612ac
  11. 21 Dec, 2015 4 commits
  12. 18 Dec, 2015 10 commits
    • Ayush Tiwari's avatar
      erp5 kernel jupyter · 8206dcdd
      Ayush Tiwari authored
      ipython_notebook SR hooked with ERP5 kernel.
      This kernel helps in interaction between erp5 and Jupyter frontend.
      The patches have been cleaned up
      
      Features:
      
      - All the code execution is being done at erp5 side, Jupyter just acts as dumb client.
      - Receives result as string and its mime_type and thanks to kernel, displays it accordingly.
      - Interactions b/w erp5 and Jupyter frontend are based on HTTP requests.
      
      Major changes:
      
      - Addition of erp5 kernel
      - Improvement in code according to guidelines(name, section name)
      - Use jinja template as instance file and make it more dynamic
      - Debugging added for ipython_notebook service.
      
      Note: The certificate authentication changed has been reverted to the previous
      one(done by creating wrapper around openssl command) for now.
      
      /cc @Tyagov
      /reviewed-by @kirr, @jerome  (on nexedi/slapos!33)
      8206dcdd
    • Kirill Smelkov's avatar
      Jupyter: --matplotlib=inline is nether supported nor needed · 837c05c6
      Kirill Smelkov authored
      @jerome added --matplotlib=inline in 48eefab5 (ipython notebook) but it is
      really neither needed:
      
         @jerome
         I remember adding this --matplotlib=inline line, but I am not sure it was
         ever needed. Using magic %matplotlib in notebook should be enough.
      
         @tiwariayush
         Yeah, for inline matplotlib in default python kernel, magics do there
         work(therefore neither pylab nor matplotlib alias are needed while starting the
         server), so I'd say leave this commit as it is and regarding version updation:
         a new patch making change wherever required.
      
      nor supported:
      
         $ cat .slappart0_ipython_notebook.log
         [W 15:51:35.454 NotebookApp] Unrecognized alias: '--matplotlib=inline', it will probably have no effect.
      
      Remove it.
      
      P.S.
      
      '--logfile' isn't available for ipython version 3.2.0 but we are not removing
      it since we are planning to upgrade IPython to versions 4.x where it is supported.
      
      Based on patch by @tiwariayush  (see nexedi/slapos!33)
      837c05c6
    • Ayush Tiwari's avatar
      Jupyter: Change section name to instance-jupyter so as not to raise conflict... · bd3d8e48
      Ayush Tiwari authored
      Jupyter: Change section name to instance-jupyter so as not to raise conflict in case of multiple extends
      
      /reviewed-by @kirr  (on !33)
      bd3d8e48
    • Ayush Tiwari's avatar
      Jupyter: Publish the serialized result · 0fb4f687
      Ayush Tiwari authored
      Maintain consistency with the slapOS SR format.
      This SR can be hooked with other SR(ex:wendelin) and its better
      to follow one way of publishing result parameters
      
      [ kirr: This essentially changes publication format to JSON:
      
          $ xslapos proxy show --params
      
          # before
          slappart0: ipython_notebook (type default)
              url = https://[2001:67c:1254:e:49::952d]:8888
              monitor_url = https://[2001:67c:1254:e:49::952d]:9685
      
          # after
          slappart0: ipython_notebook (type default)
              _ = {"url": "https://[2001:67c:1254:e:49::952d]:8888", "monitor_url": "https://[2001:67c:1254:e:49::952d]:9685"}
      
        I'm not convinced we really need this, nor that the .serialized version is
        the most oftenly used one:
      
          slapos$ git grep 'slapos.cookbook:publish$' |wc -l
          59
          slapos$ git grep 'slapos.cookbook:publish.serialised$' |wc -l
          13
      
        but we can have it and see how it goes, reverting if needed ]
      
      /cc @jerome
      /proposed-for-review-on nexedi/slapos!33
      0fb4f687
    • Ayush Tiwari's avatar
      Jupyter: Set log-level alias to 'DEBUG' for ipython notebook service · 3bf1cce9
      Ayush Tiwari authored
      This helps in logging up the requests made by ipython_notebook service
      
      [ kirr: To be clear - until log-level is set to DEBUG, IPython notebook does
        not log HTTP requests, and since logging of HTTP requests is considered normal
        for most of our services (Zope, Apache, etc), it makes sense to enable such
        functionality for notebook too.
      
        There is not much additional noise produced by --log-level=DEBUG - in
        practice ipython only prints what config files it uses on startup, so this
        should be ok to go. ]
      
      /reviewed-by @kirr, @jerome  (on !33)
      3bf1cce9
    • Ayush Tiwari's avatar
      Jupyter: Hook ERP5 Kernel · 33954c66
      Ayush Tiwari authored
      ERP5 kernel basic info/workflow:
      
      1. User enters code on notebook cell and executes
      2. Code is sent to kernel via websockets
      3. Kernel sends request to ERP5
      4. Code is executed by ERP5 and the result is returned back via request.
      5. Result is received and rendered on the notebook frontend.
      6. Other message formats such as error and status are also conveyed by the Kernel.
      
      [ kirr: in IPython notebook speak kernel is something that allows IPython
        notebook server side to talk to execution backend. ERP5 kernel is a thing that
        allows ipython notbook to talk to ERP5 (with help on-ERP5-server special bt5
        installed which accepts and executes commands).
      
        The bt5 to handle notebook calls on ERP5 side - erp5-data-notebook - is
        proposed to be merged into erp5.git on nexedi/erp5!29 ]
      
      /initially-reviewed-by @kirr, @Tyagov  (in a lot of places, last time on nexedi/slapos!33)
      33954c66
    • Ayush Tiwari's avatar
      IPython Notebook: Explicitly add environment variable around wrapper and use... · 77ffa61e
      Ayush Tiwari authored
      IPython Notebook: Explicitly add environment variable around wrapper and use ipython directory inside instance in env
      
      [ kirr: By default IPython keeps configuration and other files location in
        ~/.ipython . What this patch does is organize explicit directory in instance
        tree to keep such files  ]
      
      /reviewed-by @kirr  (on nexedi/slapos!33)
      77ffa61e
    • Ayush Tiwari's avatar
      IPython Notebook: Add dynamic-template-base section for common jinja related... · 2ff9b784
      Ayush Tiwari authored
      IPython Notebook: Add dynamic-template-base section for common jinja related file section and extend them with this section
      2ff9b784
    • Ayush Tiwari's avatar
      IPython Notebook: Convert instance to jinja2 template · fbfaaef8
      Ayush Tiwari authored
      [ kirr: to-Jinja2 conversion is required because jinja is more suitable to
        describing instances compared to buildout, because jinja2 has e.g. control
        structures ]
      
      /reviewed-by @kirr  (on nexedi/slapos!33)
      fbfaaef8
    • Kirill Smelkov's avatar
      wendelin: Adjust to recent ipython-notebook section renaming · ae1aa162
      Kirill Smelkov authored
      Commit cee110b2 (IPython Notebook: Fixing coding crimes for section
      names) changed IPython notebook section name to use '-' as word
      delimiter but forgot to update users, and this way wendelin build
      started to fail:
      
          INFO While:
          INFO   Installing.
          INFO   Getting section ipython_notebook.
          INFO Error: The referenced section, 'ipython_notebook', was not defined.
      
      Fix it. (And I've made sure with whole-tree git grep that there is no
      more ipython notebook users except wendelin in whole slapos.git so far)
      
      /reported-by @Tyagov
      /cc @tiwariayush
      /reviewed-by TrustMe
      ae1aa162
  13. 17 Dec, 2015 4 commits
  14. 16 Dec, 2015 2 commits
  15. 15 Dec, 2015 2 commits