1. 02 Feb, 2016 1 commit
  2. 01 Feb, 2016 4 commits
    • Kirill Smelkov's avatar
      erp5: ERP5 and Jupyter integrated together · 519a575d
      Kirill Smelkov authored
      This patch series teaches ERP5 software release to automatically instantiate
      Jupyter notebook web UI and tune it to connect to ERP5 by default. When Jupyter
      is enabled, it also installs on-server erp5_data_notebook bt5 (see
      nexedi/erp5!29 and nexedi/erp5@f662b5a2) which handles code execution requested
      for Jupyter.
      
      For ERP5 - for security and backward compatibility reasons - Jupyter
      instantiation and erp5_data_notebook bt5 install happen only if jupyter is
      explicitly enabled in instance parameters. The default is not to have Jupyter
      out of the box.
      
      On the other hand for Wendelin SR, which inherits from ERP5 SR, the
      default is to have Jupyter out of the box, because Wendelin SR is fresh
      enough without lots of backward compatibility needs, and Jupyter is
      usually very handy for people who use Wendelin.
      
      --------
      
      NOTE Currently erp5-data-notebook bt5 has the following limitations (see
      details on nexedi/slapos!43 and nexedi/erp5!29):
      
      - errors are not reported properly to users;
      - state is not fully saved to ZODB.
      
      the latter point means notebook works only if it is connected to Zope family
      with only 1 zope process. Hopefully this will be resolved some day.
      
      Technical overview about how the integration is done itself on slapos part and
      other notes are in 0a446263.
      
      /proposed-for-review-on nexedi/slapos!43
      519a575d
    • Douglas's avatar
      Jupyter: ERP5 kernel sends code using POST · cf117ccd
      Douglas authored
      Query strings used on GET requests have size limitations
      on servers and this causes big code cells to not be executed
      at all, returning only an Internal Server Error with no
      further explanation.
      
      /reviewed-by @kirr, @Tyagov (on nexedi/slapos!43)
      cf117ccd
    • Kirill Smelkov's avatar
      erp5: Provide pandas, scikit-learn & matplotlib out of the box · 5cd3fc51
      Kirill Smelkov authored
      Pandas and scikit-learn are handy to have for data processing which we
      are going to do more and more in ERP5 context. Matplotlib is very handy
      to have when one works with Jupyter, but also can be used by just
      backend code to generate graphs.
      
      From this point of view it makes sense to have this eggs always
      available out of the box. To do so move them from Wendelin to ERP5.
      
      /suggested-by @tiwariayush
      /reviewed-by @Tyagov (on nexedi/slapos!43)
      5cd3fc51
    • Kirill Smelkov's avatar
      ERP5 and Jupyter integrated together · 0a446263
      Kirill Smelkov authored
      This patch teaches ERP5 software release to automatically instantiate Jupyter
      notebook web UI and tune it to connect to ERP5 by default. When Jupyter is
      enabled, it also installs on-server erp5_data_notebook bt5 (nexedi/erp5!29)
      which handles code execution requested for Jupyter.
      
      For ERP5 - for security and backward compatibility reasons - Jupyter
      instantiation and erp5_data_notebook bt5 install happen only if jupyter is
      explicitly enabled in instance parameters. The default is not to have Jupyter
      out of the box.
      
      On the other hand for Wendelin SR, which inherits from ERP5 SR, the
      default is to have Jupyter out of the box, because Wendelin SR is fresh
      enough without lots of backward compatibility needs, and Jupyter is
      usually very handy for people who use Wendelin.
      
      ~~~~
      
      For integration, we reuse already established in ERP5 infrastructure, to
      request various slave instances, and request Jupyter in a way so it
      automatically tunes and connects to balancer of one of Zope family.
      
      Jupyter code itself is compiled by reusing
      software/ipython_notebook/software.cfg, and Jupyter instance code is
      reused by hooking software/ipython_notebook/instance.cfg.in into ERP5 SR
      properly (the idea to override instance-jupyter not to render into
      default template.cfg is taken from previous work by @tiwariayush).
      
      ~~~~
      
      I tested this patch inside webrunner with create-erp5-site software type and
      various configurations (whether to have or not have jupyter, to which zope
      family to connect it, etc).
      
      I have not tested frontend instantiation fully - because tests were done only
      in webrunner, but I've tried to make sure generated buildout code is valid for
      cases with frontend.
      
      NOTE the code in this patch depends erp5_data_notebook bt5 (nexedi/erp5!29) which just got merged to erp5.git recently (see nexedi/erp5@f662b5a2)
      
      NOTE even when erp5_data_notebook bt5 is installed, on a freshly installed ERP5, it
      is required to "check site consistency" first, so that initial bt5(s) are
      actually installed and erp5 is ready to function.
      
      /cc @vpelletier, @Tyagov, @klaus, @Camata, @tiwariayush, @Kreisel, @jerome, @nexedi
      /proposed-for-review-on nexedi/slapos!43
      0a446263
  3. 31 Jan, 2016 6 commits
  4. 27 Jan, 2016 1 commit
  5. 26 Jan, 2016 2 commits
  6. 25 Jan, 2016 2 commits
  7. 24 Jan, 2016 1 commit
    • Kirill Smelkov's avatar
      wendelin: Remove bt5_list leftover · 9aa86470
      Kirill Smelkov authored
      @rafael added this in 971d0bb7 (erp5: Make possible extent the list of
      initial business templates to install), but we dropped that change while
      merging erp5-cluster to master - see:
      
          6bbb61a8 "Merge branch 'master' into erp5-cluster", and
          e84d5e83 "Merge branch 'erp5-cluster'"
      
      6bbb61a8 claimed that it
      
          Dropped commit 971d0bb7
          ("erp5: Make possible extent the list of initial business templates to install").
      
      but it actually dropped changes only under stack/erp5/ , not software/wendelin/
      
      Fix it.
      
      /cc @rafael, @jm, @Tyagov
      /reviewed-by TrustMe
      9aa86470
  8. 22 Jan, 2016 1 commit
  9. 21 Jan, 2016 1 commit
  10. 20 Jan, 2016 3 commits
  11. 17 Jan, 2016 18 commits
    • Kirill Smelkov's avatar
      GitLab Software Release - first version · 866d9051
      Kirill Smelkov authored
      Hello up there,
      
      Here comes SlapOS port of GitLab. We start from GitLab 8.2.X as that is what we
      currently run on KVM on lab.nexedi.com, so that our data can be
      straightforwardly migrated.
      
      The SR compiles all needed software and organized all (sub-)services in one
      partition and interconnects them with unix sockets for security and speed
      reasons (see patch "gitlab: Make a plan to base instance layout on
      gitlab-omnibus and to interconnect all internal services").
      
      Services configuration files are originally taken from omnibus-gitlab
      "distribution" and incrementally ported to slapos variant. This way we
      establish a (imho) good path on how to track upstream changes and minimize our
      delta & effort supporting it.
      
      GitLab itself is patched (above patches that were already applied by upstream):
      
        - to support HTTP(S) only
        - to show site's ICP number
        - to speedup raw blob downloading ~ 17x times
          ( see patch "gitlab: Optimize raw blob downloading" for details and
            https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17 )
      
      Overall it should work and we should finally be able to migrate slapos.git
      (because of raw blob downloading is not slow now) to GitLab and all other
      Nexedi git repositories.
      
      Thanks,
      Kirill
      
      P.S.
      
      Somewhat outdated, but this picture on GitLab architecture might help to
      understand how parts are glued together:
      
      https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/architecture.md
      
      P.P.S.
      
      Native resiliency is not implemented yet, but we should be able to use gitlab
      inside resilient webrunner already.
      
      /proposed-for-review-on nexedi/slapos!39
      /partly-reviewed-by @kazuhiko, @jerome, @Yanni, @jp
      /cc @rafael, @jm
      866d9051
    • Kirill Smelkov's avatar
      gitlab: First SR version works - freeze md5 sums · 729be3b8
      Kirill Smelkov authored
      We've reached a state where first gitlab SR version should work. So as
      promised let's freeze the md5 checksums.
      
      All later patches should update corresponding md5 info when they change
      a file.
      
      /cc @kazuhiko, @jerome
      729be3b8
    • Kirill Smelkov's avatar
      gitlab: Optimize raw blob downloading · a913c2e4
      Kirill Smelkov authored
      In slapos we do a lot of automated software rebuild constantly, and thus
      there is constant flow of requests to get raw blobs from git service,
      e.g. like this
      
          https://lab.nexedi.com/nexedi/slapos/raw/master/software/wendelin/software.cfg
      
      A lot of requests comes to slapos.git repository and currently gitlab,
      out of the box, cannot keep up with that load.
      
      I've prepared patches to offload raw blobs download requests handling
      from unicorn (ruby) to gitlab-workhorse (go), and that resulted in ~ 17x
      speedup - e.g. previously our std shuttle can handle ~ 70 raw-blob
      requests/s and with my changes it is now ~ 1200 requests/s.
      
      The patches were sent upstream
      
          https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17
      
      and we discussed with GitLab people and made a plan how to proceed
      incrementally. It will probably take some time for gitlab team to fully
      accept the approach though.
      
      For now we can use our gitlab-workhorse fork. The patches itself are:
      
          kirr/gitlab-workhorse@1b274d0d
          kirr/gitlab-workhorse@2beb8c95
      
      /cc @kazuhiko, @jerome, @jm
      a913c2e4
    • Kirill Smelkov's avatar
      gitlab: Switch to "GitLab Nexedi Edition" · 74d4ea62
      Kirill Smelkov authored
      GitLab Nexedi Edition is currently upstream 8.2.X + the following
      patches:
      
          - HTTP(S) is made to be default clone protocol
      
              kirr/gitlab-ce@5c1f2fb3
      
            and SSH info is completely removed from UI
      
              kirr/gitlab-ce@dfe9fb16
              kirr/gitlab-ce@f3f84743
      
            so essentially the only way to access a repository is via HTTP(S).
      
          - Rake check tasks are adjusted to exit with non-zero code if there
            is a failure
      
              kirr/gitlab-ce@a93ae418
      
            We need this for promises to work correctly with failures being
            detected, not silently skipped. The patch was sent upstream:
      
              https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1885
      
          - GitLab supports setting up site's ICP License in gitlab.yml and
            shows it in appropriate places together with info about GitLab
            itself:
      
              kirr/gitlab-ce@e7e0fd88
              kirr/gitlab-ce@79c127e6
      
          + other cosmetic/minor changes.
      
      More patches will probably come (e.g. apply a single patch from a
      merge-request with `git am` without creating merge commit for just 1
      patch, etc) but for now that's all.
      
      NOTE ICP is non-ascii text with hieroglyphs. slapos.core was taught to
          be able to pass parameters with non-ascii values to instance:
      
              slapos.core@347d33d6
      
          That patch is included in slapos.core 1.3.15, but as we currently
          have a lot of older slapos.core deployed (e.g. 1.3.5 on my
          development webrunner) a workaround is (hopefully temporarily) used
          to pass non-ascii values as URL-encoded strings.
      
      /cc @kazuhiko, @jerome, @rafael
      74d4ea62
    • Kirill Smelkov's avatar
      gitlab: Publish connection URL · 3c445ad3
      Kirill Smelkov authored
      In the previous patch we setup nginx service which listens to the world
      and as such gitlab service becomes to be ready to used - so publish
      backend URL.
      
      NOTE we'll need to optimise and tweak gitlab a bit further in upcoming
          patches, so it can be really used under load and with our use-cases,
          but even now it listens to http ok and generally works.
      
      /cc @kazuhiko, @jerome
      3c445ad3
    • Kirill Smelkov's avatar
      gitlab/nginx: Slapos'ify config and turn nginx into a service · 85f7d7e3
      Kirill Smelkov authored
      Go through nginx configuration templates and convert them to jinja2 with
      slapos parameters (reminder: names and default values are imported from
      omnibus-gitlab 8.2.3+ce.0-0-g8eda093), except commenting out features we
      do not want to support (yet ?).
      
      As nginx is a reverse-proxy, i.e. it integrates all internal services
      and works as frontend to them, our gitlab service is now ready to listen
      and talk to the world over (standard to slapos services backend) IPv6.
      
      Nginx also acts as SSL termination point - for it to work by default we
      setup self-signed certificate for the backend, which can be manually
      changed to proper certificate if needed. Backend certificate is used
      if gitlab is configured to work in HTTPS mode (and frontend certificate
      is another story).
      
      NOTE ssl certificate is generated with just `openssl req ...` - yes, there
          is slapos.cookbook:certificate_authority.request but it requires
          to start whole service and has up to 60 seconds latency to generate
          certificate. And we only need to run 1 command to do that...
      
      The features disabled are:
      
          - http -> https redirection
      
            not needed for us at nginx level - the frontend can do the
            redirection and also gitlab speaks HSTS on https port so when we access
            https port via http protocol, it gets redirected to https.
      
          - kerberos
          - ssl_dhparam
          - providing custom nginx configuration via instance parameter
      
      /cc @kazuhiko, @jerome
      85f7d7e3
    • Kirill Smelkov's avatar
      gitlab: Hook nginx configuration files into SR system · 45127f6d
      Kirill Smelkov authored
      Like with Rails configuration files, hook nginx configuration files into
      SR / instance build process; rename *.erb -> *.in and add our header.
      
      The templates are still not valid - a lot of erb code is left there -
      we'll slapos'ify it incrementally in the following patches.
      
      /cc @kazuhiko, @jerome
      45127f6d
    • Kirill Smelkov's avatar
      gitlab: Import nginx http configuration from omnibus-gitlab · 61544d87
      Kirill Smelkov authored
      Like with Rails configuration this first step is pristine import of
      nginx configuration files from omnibus-gitlab. All files were imported
      as-is in their ERB form and filenames from omnibus-gitlab
      8.2.3+ce.0-0-g8eda093 from here:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default
      
      We import only nginx main http configuration - nginx's CI and Mattermost
      configurations are not imported, as we do not support CI and Mattermost (yet ?).
      
      As with Rails configuration files, we will convert the templates to
      jinja2 and adjust them to slapos version in the following patches.  We
      will also use the same (commit from last-erb-mod commit + merge)
      approach to track upstream changes.
      
      /cc @kazuhiko, @jerome
      61544d87
    • Kirill Smelkov's avatar
      gitlab: Setup sidekiq service · 4c127fdd
      Kirill Smelkov authored
      Sidekiq[1] is used in GitLab as background jobs manager - i.e. if a
      request handler needs to spawn some non-light job - it adds it to
      sidekiq queue (in Redis) and relies on sidekiq service to later pick
      this job up and execute it.
      
      The service is setup with just to run bin/gitlab-sidekiq with
      appropriate queues (extracted from omnibus-gitlab) and appropriate
      settings to controlling GitLab's sidekiq Out-Of-Memory killer[2].
      
      NOTE Unlike unicorn OOM killer, Sidekiq memory killer just makes sidekiq
          processes to be SIGKILL terminated and relies on managing service to
          restart it. In slapos we don't have mechanism to set autorestart=true,
          nor bang/watchdog currently work with slapproxy, so we setup to do
          such monitoring ourselves manually with here-introduced
          watcher-sigkill program.
      
      NOTE2 sidekiq promise, because it is rake/gitlab based, is slow to
          load/run and thus is put into etc/promise.slow/
      
      [1] http://sidekiq.org/
      [2] https://gitlab.com/gitlab-org/gitlab-ce/blob/1322bd78/doc/operations/sidekiq_memory_killer.md
      
      /cc @kazuhiko, @jerome
      4c127fdd
    • Kirill Smelkov's avatar
      gitlab: Upgrade gitlab-shell & gitlab-workhorse to versions which propagate $HOME · 76e371cd
      Kirill Smelkov authored
      As was described in the previous patch, we need $HOME to be propagated
      by this programs so that git can find partition's .gitconfig.
      
      Specifically we need the following patches to be present in our build:
      
          https://gitlab.com/gitlab-org/gitlab-shell/commit/9e087f64
          https://gitlab.com/gitlab-org/gitlab-workhorse/commit/b5f1b803
      
      They both have been applied upstream very close to revisions we
      previously had in software.cfg, so we only need to update the revisions
      to get them.
      
      /cc @kazuhiko, @jerome
      76e371cd
    • Kirill Smelkov's avatar
      gitlab: Slapos'ify / tweak gitconfig and hook it into the system · 264d785a
      Kirill Smelkov authored
      Convert gitconfig template to jinja2 (reusing already-there
      `email_display_name` and `email_from` parameters for commits generated by
      gitlab).
      
      System-level git config from gitlab-omnibus is also imported to this
      file (on slapos we cannot tweak system-level git config -
      software/.../parts/git/... is read-only for programs in instance
      partitions - so we move all gitlab's system-wide git settings to this
      "user-level" gitconfig.
      
      System gitconfig in omnibus is defined here:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda093/files/gitlab-cookbooks/gitlab/attributes/default.rb#L23
      
      so it is
      
          pack.threads = 1    and
          receive.fsckObjects = true
      
      which makes sense to not waste a lot of memory when packing and not to
      allow corrupt objects to enter to system by evil users intentionally.
      
      To make the file foundable by git - we put it into partition root
      directory and set $HOME to point to partition root when running
      appropriate programs / services.
      
      NOTE we'll need to upgrade gitlab-shell and gitlab-workhorse to
          propagate $HOME for this setting to actually have effect.
          See the next patch.
      
      /cc @kazuhiko, @jerome
      264d785a
    • Kirill Smelkov's avatar
      gitlab: Import gitconfig from omnibus-gitlab · 8f945bd2
      Kirill Smelkov authored
      Like with Rails configuration files, this is pristine import of template
      gitconfig from omnibus GitLab from
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default
      
      This is only a "user" part of git configuration. System-wide
      configuration is generated dynamically:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda093/files/gitlab-cookbooks/gitlab/attributes/default.rb#L23
      
      and we'll import it by hand in the follow-up patches.
      
      /cc @kazuhiko, @jerome
      8f945bd2
    • Kirill Smelkov's avatar
      gitlab: Setup gitlab-workhorse service · 6ddc1c30
      Kirill Smelkov authored
      Gitlab-workhorse[1] is a service which offloads Ruby-on-Rails based
      GitLab from long-running and slow requests. It is written in Go.
      
      Now as we have unicorn service set up, we can setup gitlab-workhorse
      service (which uses unicorn as authentication backend).
      
      Gitlab-workhorse setup is easy - it is just one program and several
      command line options to point to unicorn socket and to configure on
      which unix socket gitlab-workhorse will listen itself.
      
      NOTE we have to care that git and ruby to be on PATH when running
          gitlab-workhorse - because on e.g. git push'ing workhorse will run `git
          receive-pack` and a hook will be called which calls gitlab-shell,
          which is written in ruby.
      
      NOTE2 promise to check whether gitlab-workhorse is alive is to ping it
          via URL to non-existent endpoint and check for proper 403 HTTP code
          returned.
      
      [1] https://gitlab.com/gitlab-org/gitlab-workhorse
      
      /cc @kazuhiko, @jerome
      6ddc1c30
    • Kirill Smelkov's avatar
      gitlab: Compile assets on instantiation and make sure DB is properly... · 5a744de7
      Kirill Smelkov authored
      gitlab: Compile assets on instantiation and make sure DB is properly setup/migrated before unicorn runs
      
      There are several actions that needs to be done on gitlab instance
      upgrade:
      
          - we have to (re-)compile assets
          - we have to migrate DB
      
      and also before the first run
      
          - we have to initialize DB
      
      We can compile assets as part of instantiation process, but regarding
      DB migration / setup - it is not currently possible to do that as part
      of instantiation - for that operations we need PG & Redis to be already
      running, but the first time slapos instantiates an SR it first prepares
      all services, and only after instantiation is done, starts them all. There is
      currently no way to hook into starting process, and run some scripts
      after one service is started but before another service startup...
      
      So the solution is: to perform such actions in delayed mode as part of
      application - unicorn service - startup: it makes sure PG is running and
      initializes it and does other actions which needs to be done to migrate
      the DB. Only if/after they succeed the main application is started.
      
      NOTE the comment about unicorn/gitlab startup slowness from the previous
          patch still holds true - so in order to get "all ok" after
          instantiation, it is required to perform the instantiation several
          times, because unicorn promise initially fails.
      
      /cc @kazuhiko, @jerome
      5a744de7
    • Kirill Smelkov's avatar
      gitlab: Set up unicorn service · 481e4758
      Kirill Smelkov authored
      Now that all gitlab Rails application configuration files are ready, we
      can setup unicorn service to start it.
      
      NOTE there is a promise to check unicorn by url which works, but there
          are also rake tasks to check gitlab itself, e.g. like
      
              gitlab:app:check
              gitlab:gitlab_shell:check
              gitlab:repo:check
      
          Unfortunately this tasks are slow to run (and gitlab:repo:check is
          very slow to run). That's why we do not put them into etc/promise/ -
          if we do - slapos reports promises time outs.
      
          What we do is we put them into etc/promise.slow/ so we have those
          scripts ready, but currently no one automatically checks them.
      
          Again, the promise to check unicorn just by accessing it by URL is
          there and is checked automatically out of the box.
      
      NOTE2 GitLab is very slow to load. That's why it can take some time
          after unicorn starts that it's promise start to report ok. This can
          show itself as temporary instantiation errors which say promise such and
          such failed.
      
      NOTE3 Unicorn start, but so far we did not cared to setup GitLab DB
          schema on instantiation. That's why unicorn remains not very usable
          and a lot of requests fail. We'll teach instance to setup DB and
          perform all other needed settings in the next patch.
      
      /cc @kazuhiko, @jerome
      481e4758
    • Kirill Smelkov's avatar
      gitlab: Add helper to set up promise to check something via url · 2772191c
      Kirill Smelkov authored
      Like with [promise-wrapper] a recipe could do
      
          [promise-<service>]
          <= promise-byurl
          url     = ...
      
      and a script to check such ur will be generated and automatically put
      into etc/promise/<service>.
      
      /cc @kazuhiko, @jerome
      2772191c
    • Kirill Smelkov's avatar
      gitlab/unicorn: Automatically load all available CPUs by default · 5dc6321c
      Kirill Smelkov authored
      Automatically configure unicorn to spawn as much worker processes as
      there are CPUs on the system by default.
      
      GitLab omnibus pre-hardcodes this value default to 2 (which we copied)
      and then also tweaks it this way in active code
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/attributes/default.rb#L230
      
      which we also do here.
      
      /cc @kazuhiko, @jerome
      5dc6321c
    • Kirill Smelkov's avatar
      gitlab/gitlab-shell-config.yml: Explicitly point it to secret file · b55d823d
      Kirill Smelkov authored
      Explicitly point gitlab-shell to location where we keep secrets.
      
      We already pointeg gitlab to that place and now we do that for
      gitlab-shell so those 2 peieces can connect to each other ok.
      
      Regarding the setting itself - there is no such block in omnibus-gitlab,
      but it is present in gitlab-shell configuration example:
      
          https://gitlab.com/gitlab-org/gitlab-shell/blob/82b3a4e8/config.yml.example#L35
      
      /cc @kazuhiko, @jerome
      b55d823d