1. 17 Jan, 2016 16 commits
    • Kirill Smelkov's avatar
      gitlab/unicorn.rb: First round of slaposification · 0aae33d9
      Kirill Smelkov authored
      Convert unicorn parameters to slapos and configure it to listen on unix
      socket only.
      
      ( Omnibus configures unicorn to listen on unix socket and
        loopback TCP, mainly because gitlab-shell could not connect to unicorn
        via unix socket until recently:
      
            https://gitlab.com/gitlab-org/gitlab-shell/commit/184385ac
      
        But as it can now, there is no point to keep on TCP port open )
      
      To be able to do such configuration we add stub to unicorn service
      section (to create needed directories where to keep the socket).
      
      There will be follow-up patch which configures unicorn pre/post-forking
      actions, which is not trivial and thus better be done on its own.
      
      /cc @kazuhiko, @jerome
      0aae33d9
    • Kirill Smelkov's avatar
      gitlab/gitlab.yml: Slapos'ify rest of it · c3f1f0a9
      Kirill Smelkov authored
      Convert the rest of this configuration file to slapos.
      
      It is straightforward conversion of parameters except:
      
          - access-via-ssh is disabled (gitlab slapos version does not support
            ssh access and supports HTTP(S) only by design on purpose)
      
          - we do not support restricting possible projects visibility via
            instance parameter (very low chance this will be needed in
            practice)
      
          - default issue-closing pattern is just ok for now and not
            configurable
      
          - support for builds, build artifacts & CI is disabled (we do not
            support CI (yet ?))
      
          - some internal defaults are just ok (e.g. where to organize
            directory for keeping repositories archives for downloads)
      
          - reply-by-email is not supported (yet ?)
      
          - we do not support LFS (yet ?) - just plain git hosting is ok for now.
      
          - Gravatar defaults are ok for now and not configurable.
      
          - Support for LDAP is disabled
      
          - Support for Kerberos is disabled
      
          - Support for OmniAuth is disabled
      
          - Satellites path is just /dev/null as we start from version where
            satellites are already non-existent.
      
          - Uploading backups to somewhere via GitLab's builtin mechanism is
            not supported - we'll use SlapOS native backup and resiliency for
            this.
      
          - Support for Google analytics is disabled.
      
          - Support for Piwik is disabled.
      
          - we are ok (for now) with default rack-attack git settings
      
      /cc @kazuhiko, @jerome
      c3f1f0a9
    • Kirill Smelkov's avatar
      gitlab: Determine current slapuserX in instance · 34419064
      Kirill Smelkov authored
      This user will need to be specified several times in configuration
      files, as by default gitlab uses 'git' user and does "sudo" to it if it
      is not current.
      
      We will use {{ backend_info.user }} in the upcoming patches.
      
      /cc @kazuhiko, @jerome
      34419064
    • Kirill Smelkov's avatar
      gitlab/gitlab.yml: Handle "external URL" · 93362a08
      Kirill Smelkov authored
      GitLab has a notion of "external URL" - the canonical "frontend" URL the
      server is reachable through: this URL is used as prefix to show
      e.g. git-clone URL for repositories, etc, even if a server can be
      reachable via several frontends.
      
      Add external_url handling to slapos instance.
      
      NOTE whether to use https or not is also defined by external_url, in
      particular by external_url scheme.
      
      /cc @kazuhiko, @jerome
      93362a08
    • Kirill Smelkov's avatar
      gitlab/smtp_settings.rb: Convert/integrate to slapos · c64f7ece
      Kirill Smelkov authored
      Convert to slapos SMTP settings for gitlab:
      
          - convert to jinja2
          - remove support for gitlab CI (we do not support it (yet ?))
          - add handling of `smtp_enable` parameter directly to that file
            ( omnibus handles this parameter externally and just removes
              smtp_settings.rb if it is true )
      
      NOTE smtp_settings.rb contains SMTP password, so it is mode is set to 0600.
      
      /cc @kazuhiko, @jerome
      c64f7ece
    • Kirill Smelkov's avatar
      gitlab/rack_attack.rb: Convert/integrate to slapos · a44f5a43
      Kirill Smelkov authored
      Just another 2 simple parameters (attack detection tunables) conversion
      to jinja2/slapos.
      
      /cc @kazuhiko, @jerome
      a44f5a43
    • Kirill Smelkov's avatar
      gitlab/config.ru: Convert/integrate to slapos · 41b1edb5
      Kirill Smelkov authored
      Just convert 2 parameters used in that file to jinja syntax and add
      those parameters (unicorn OOM killer tunables) to gitlab-parameters.cfg
      
      /cc @kazuhiko, @jerome
      41b1edb5
    • Kirill Smelkov's avatar
      gitlab/resque.yml: Tweak to integrate gitlab with internal redis · b20c258b
      Kirill Smelkov authored
      A simple change just to point resque to redis unix socket.
      
      /cc @kazuhiko, @jerome
      b20c258b
    • Kirill Smelkov's avatar
      gitlab/database.yml: Tweak to integrate gitlab with internal postgresql · a73d20f4
      Kirill Smelkov authored
      We tweak database.yml to point to our postgresql unix socket; set
      adapter to hardcoded postgresql, encoding to unicode and omit collation
      (which according to omnibus-gitlab is used for mysql only).
      
      The only instance parameter imported from omnibus is `db_pool` - how
      many connection to a DB to keep open in a RoR thread/process.
      
      XXX we use db's superuser as a user to connect. Is it ok to do even if
          the whole DB is used only for gitlab? (I think it is ok for the
          first iteration, but we'll probably need to refine this later)
      
      /cc @kazuhiko, @jerome
      a73d20f4
    • Kirill Smelkov's avatar
      gitlab: Introduce macro library · 2e81276b
      Kirill Smelkov authored
      Introduce a library of Jinja2 macros that will be handy to use in
      templates. For now we add only 2 macros:
      
          cfg(name)   - to get instance configuration parameter `name`,   and
          cfg_bool    - to get truth value of ----//----
      
      The reason we introduce cfg() is that we will need to use a lot of
      parameters in many places and it is much more handy to write, e.g.
      
          cfg('email_enabled')
      
      compared to
      
          instance_parameter:configuration.email_enabled
      
      /cc @kazuhiko, @jerome
      2e81276b
    • Kirill Smelkov's avatar
      gitlab: Organize place to keep parameters & their default imported from gitlab-omnibus · de860ba5
      Kirill Smelkov authored
      We will be using a several dozens of parameters to control gitlab
      instance. It makes sense not to deviate in such parameters namings and
      defaults from omnibus version.
      
      Thus for such parameters - for clarity - we organize a separate file
      where we will be keeping them - gitlab-parameters.cfg.
      
      In this patch series all used parameters will be "imported" from
      omnibus-gitlab 8.2.3+ce.0-0-g8eda093.
      
      NOTE it is maybe better to try to autogenerate that file from upstream
          omnibus parameters definitions. If time will tell it becomes hard to
          maintain our copy - we'll consider going that way.
      
      /cc @kazuhiko, @jerome
      de860ba5
    • Kirill Smelkov's avatar
      gitlab: Hook gitlab- and gitlab-shell- configuration files into the system · 13169cab
      Kirill Smelkov authored
      - Download them on SR build and pass info to instance
      - Instance prepares to process them as jinja2 templates
      - Instance hooks the files into configuration location as appropriate
      
      Every file so far is renamed *.erb -> *.in and a header added showing
      that this file is autogenerated with links about what was the base
      gitlab and/or omnibus version and omnibus reference revision this
      template was last updated for.
      
      So far all result configuration files are invalid - because ERB syntax
      is there. We will convert the configuration files to proper jinja2
      syntax and to using slapos parameters incrementally in the upcoming
      patches.
      
      NOTE (again): md5 sums are not yet fixed - we will fix them in the end
          of gitlab patches series after applying all tweaking changes.
      
      /cc @kazuhiko, @jerome
      13169cab
    • Kirill Smelkov's avatar
      gitlab: Import gitlab-ce & gitlab-shell configs from omnibus-gitlab · 6fd7b987
      Kirill Smelkov authored
      Pristine import of template configuration files from omnibus GitLab
      package. All files were imported as-is in their ERB form and filenames
      from omnibus-gitlab 8.2.3+ce.0-0-g8eda093 from here:
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default
      
      We will convert the templates to jinja2 and adjust them to slapos
      version in the following patches.
      
      Scheme for synchronizing with future upstream changes is envisioned as this:
      
          - checkout latest commit which updated pristine erb files
          - copy updated files from omnibus-gitlab, and commit the updates
          - checkout slapos master
          - merge commit that updated erb
      
      That should reasonably work with not too-many conflicts and even those
      should be not hard to resolve (with `git mergetool` e.g. in kdiff3)
      
      /cc @kazuhiko, @jerome
      6fd7b987
    • Kirill Smelkov's avatar
      gitlab: Organize per-instance gitlab work tree · 2ddc5b0c
      Kirill Smelkov authored
      Organize per-instance place for gitlab configuration and work directory.
      
      Unfortunately as GitLab is Ruby-on-Rails application, it is not possible
      to keep its code in one place and have multiple separate configuration
      sets in different places and start that code for a configuration set -
      GitLab and Rails insist to get configuration from relative to source
      code tree.
      
      GitLab omnibus "solves" this by having only one configuration set and
      having symlinks from code to that only configiration set. In slapos we
      can potentially have several instances for one software and thus we
      cannot do that.
      
      With such limitations a proper solution would be to bind-mount software
      code into instance filesystem namespace close to configuration - that
      way the code will be only one and will find proper per-instance config.
      Currently we do not have namespaces available on slapos unfortunately,
      thus something else is needed.
      
      The workaround I decided to do is this: to clone cloned gitlab
      repository from software/ space to instance/ space and adjust it in
      instance space. This has the following drawbacks:
      
          - code is duplicated
          - code becomes read-write, instead of being read-only
      
      but imho it is the most practical thing to do. Another solution could be
      to patch GitLab / Rails to remove "config lives in code" assumption, but
      the number of places where this needs to be done is really many.
      
      NOTE gems which gitlab uses and which were installed during software
          compilation are not duplicated - they are reused via bundler - via
          pointing BUNDLE_GEMFILE to original location in software.
      
      NOTE2 For instance tasks and also for maintanace convenience we establish
          <instance>/bin/gitlab-* programs, e.g. gitlab-rake, which e.g. for
          gitlab-rake will run rake with correctly loaded gitlab environment -
          like in gitlab-omnibus.
      
      /cc @kazuhiko, @jerome, @jp
      2ddc5b0c
    • Kirill Smelkov's avatar
      gitlab: Redis service · 0d286c5d
      Kirill Smelkov authored
      Organize internal Redis service, like with PostgreSQL in the previous
      patch, with the help of slapos.cookbook:redis.server recipe.
      
      Like with postgresql, and as we planned, redis listens only on
      internal-to-partition unix socket.
      
      The recipe establishes both service and promise to check it is alive;
      we only need to setup log rotation manually.
      
      /cc @kazuhiko, @jerome
      0d286c5d
    • Kirill Smelkov's avatar
      gitlab: PostgreSQL service · 470719fe
      Kirill Smelkov authored
      Organize internal PostgreSQL database which will be used as DB for
      Roby-on-Rails GitLab and listens only on unix socket (for security and
      performance reasons - see earlier intro patch).
      
      To do it we use slapos.cookbook:postgres recipe, with disabling
      "listen-to-network" via passing empty sets to ipv4 and ipv6 recipe
      arguments.
      
      The promise to check whether DB is alive is just `psql -c '\q'` which
      will error if failing to connect to DB, but exit silently if connected ok.
      
      Explicit log rotation is not needed - as postgresql logs to
      stdout/stderr - not to a file - logs are handled by slapos - put into
      .slappartX_postgresql.log and automatically rotated there.
      
      XXX omnibus-gitlab tunes postgresql with shared_buffers and other
      parameters, most likely for performance reasons - see e.g.
      
          https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-2-stable/files/gitlab-cookbooks/gitlab/templates/default/postgresql.conf.erb#L113
      
      I decided not to fine-tune postgresql for now, and get on-field feedback
      first, and then, if needed, we can tune.
      
      /cc @kazuhiko, @jerome
      470719fe
  2. 07 Jan, 2016 2 commits
    • Kirill Smelkov's avatar
      gitlab: Add helper for setting up promises · 5e4a181a
      Kirill Smelkov authored
      A recipe could do
      
          [promise-<service>]
          <= promise-wrapper
          command-line = ...
      
      and the wrapper will be put automatiaclly into etc/promise/<service>.
      
      ( for this to happen !py! magic is used again, like we did for logrotate
        and cron entries before )
      
      /cc @kazuhiko, @jerome
      5e4a181a
    • Kirill Smelkov's avatar
      gitlab: Make a plan to base instance layout on gitlab-omnibus and to... · e7c5c05a
      Kirill Smelkov authored
      gitlab: Make a plan to base instance layout on gitlab-omnibus and to interconnect all internal services via unix sockets
      
      Upcoming changes will follow two points:
      
      - we try to base our gitlab setup on how it is done in
        gitlab-omnibus[1] with the idea to ease tracking upstream changes to
        instance setup.
      
      - we will interconnect all internal services via unix sockets only.
      
        The reason to do it is twofold:
      
          1. easier security: currently files on different slapos partitions
             are isolated from each other, but there is no "in-between-partitions"
             networking isolation - thus (potentially evil) programs can
             access internal services on other slapos partition.
      
             permissions to access unix sockets, on the other hand, are
             managed by filesystem-level permissions, and thus unix sockets in
             one partition will be, by default, isolated from programs on
             another partitions.
      
          2. It is well known that UNIX sockets are faster than TCP over
             loopback. For example for our std shuttles they have 2 times lower
             latency and ~ 2-3 times more throughput compared to TCP over loopback
      
          More details on 1 & 2 can be found e.g. here:
      
          nexedi/slapos!27
          https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/30
      
      /cc @kazuhiko, @jerome
      
      [1] https://gitlab.com/gitlab-org/omnibus-gitlab
      e7c5c05a
  3. 06 Jan, 2016 2 commits
    • Kirill Smelkov's avatar
      gitlab: Add empty instance · ab6d2f28
      Kirill Smelkov authored
      Add stub instance configuration which just establishes a way to have
      several software types(*), pass all needed info from software to
      instance, organizes base directory and establishes log rotation base for
      upcoming services.
      
      Log rotation is done with the help of cron periodicallly calling
      logrotate. The rotation is done in "copytruncate" mode - i.e. log file
      is not moved away and signal sent for service to reopen it, but instead
      log content is just copied to outside and there is no need for a service
      to reopen it's log file.
      
      The reason it is done this way, is that there is a chance of not
      handling such "reopen-log-file" callbacks correctly on a service side,
      and so the net is full of crashing reports, e.g. like this:
      
          http://serverfault.com/questions/627521/why-is-logrotate-causing-apache-to-seg-fault-each-time
      
      That's why we take a safer approach instead, even if "copytruncate" mode
      is risking to loose several log entries(**) on rotation.
      
      NOTE services will organize log rotation with just
      
          [logrotate-entry-<service>]
          <= logrotate-entry
          log     = path/to/log/files/*.log
      
      For this to work some "!py!" magic (our way to serialize object into
      executable python and process it in buildout recipes) is used to process
      section names.
      
      The approach trick is also used for cron, e.g. logrotate registers to
      cron this way:
      
          [cron-entry-logrotate]
          <= cron-entry
          time    = daily
          command = ${logrotate:wrapper}
      
      NOTE2 instance md5 are not fixed yet - we'll fix them after applying all
          patches in gitlab series.
      
      (*) for now there is only 1 - "gitlab", but we'll need to have "-export"
          and "-import" for resiliency in the future.
      
      (**) ideally such things should be done with logfs - a filesystem
          specializeing in logging - for client services it will look like as
          they just continue to write to log file, and on log service side, the
          rotation can happen, all transparent to client service.
      
      /cc @kazuhiko, @jerome
      ab6d2f28
    • Kirill Smelkov's avatar
      Start of GitLab Software Release · 5e971c58
      Kirill Smelkov authored
      First step - build all needed software. We build:
      
      - Git
      - PostgreSQL 9.2
      - Redis 2.8
      - Nginx
      
      - gitlab-shell
      - gitlab-workhorse
      - gitlab-ce 8.2 itself
      
      and everything which is needed to build the above programs.
      
      Git is needed because GitLab is a git-hosting service and uses git
      underneath. PostgreSQL is used as DB by gitlab and Redis as a cache.
      
      GitLab-shell is a small project to manage ssh access to the service
      (we'll disable ssh though) and to perform all "change a repository"
      operations.
      
      GitLab-workhorse is a service which offloads long-running or slow
      request from main GitLab service.
      
      GitLab-ce is the main Ruby-on-Rails-based web application.
      
      Ruby- and Go- based programs are built in a way similar to:
      
          - 31a45a94    (helloworld & helloweb: Ruby version), and
          - 24e82414    (helloworld & helloweb: Go version)
      
      Version of all components, except Git, were picked the same, as used by
      gitlab omnibus v8.2 .
      
      /cc @kazuhiko, @jerome
      5e971c58
  4. 04 Jan, 2016 1 commit
  5. 28 Dec, 2015 4 commits
  6. 26 Dec, 2015 1 commit
  7. 23 Dec, 2015 1 commit
    • Julien Muchembled's avatar
      re6stnet: update and cleanup · 284612ac
      Julien Muchembled authored
      ../../stack/slapos.cfg is removed from component/*/buildout.cfg
      because we normally don't specify it in component/
      The OBS package will need to extend it.
      284612ac
  8. 21 Dec, 2015 4 commits
  9. 18 Dec, 2015 9 commits
    • Ayush Tiwari's avatar
      erp5 kernel jupyter · 8206dcdd
      Ayush Tiwari authored
      ipython_notebook SR hooked with ERP5 kernel.
      This kernel helps in interaction between erp5 and Jupyter frontend.
      The patches have been cleaned up
      
      Features:
      
      - All the code execution is being done at erp5 side, Jupyter just acts as dumb client.
      - Receives result as string and its mime_type and thanks to kernel, displays it accordingly.
      - Interactions b/w erp5 and Jupyter frontend are based on HTTP requests.
      
      Major changes:
      
      - Addition of erp5 kernel
      - Improvement in code according to guidelines(name, section name)
      - Use jinja template as instance file and make it more dynamic
      - Debugging added for ipython_notebook service.
      
      Note: The certificate authentication changed has been reverted to the previous
      one(done by creating wrapper around openssl command) for now.
      
      /cc @Tyagov
      /reviewed-by @kirr, @jerome  (on nexedi/slapos!33)
      8206dcdd
    • Kirill Smelkov's avatar
      Jupyter: --matplotlib=inline is nether supported nor needed · 837c05c6
      Kirill Smelkov authored
      @jerome added --matplotlib=inline in 48eefab5 (ipython notebook) but it is
      really neither needed:
      
         @jerome
         I remember adding this --matplotlib=inline line, but I am not sure it was
         ever needed. Using magic %matplotlib in notebook should be enough.
      
         @tiwariayush
         Yeah, for inline matplotlib in default python kernel, magics do there
         work(therefore neither pylab nor matplotlib alias are needed while starting the
         server), so I'd say leave this commit as it is and regarding version updation:
         a new patch making change wherever required.
      
      nor supported:
      
         $ cat .slappart0_ipython_notebook.log
         [W 15:51:35.454 NotebookApp] Unrecognized alias: '--matplotlib=inline', it will probably have no effect.
      
      Remove it.
      
      P.S.
      
      '--logfile' isn't available for ipython version 3.2.0 but we are not removing
      it since we are planning to upgrade IPython to versions 4.x where it is supported.
      
      Based on patch by @tiwariayush  (see nexedi/slapos!33)
      837c05c6
    • Ayush Tiwari's avatar
      Jupyter: Change section name to instance-jupyter so as not to raise conflict... · bd3d8e48
      Ayush Tiwari authored
      Jupyter: Change section name to instance-jupyter so as not to raise conflict in case of multiple extends
      
      /reviewed-by @kirr  (on !33)
      bd3d8e48
    • Ayush Tiwari's avatar
      Jupyter: Publish the serialized result · 0fb4f687
      Ayush Tiwari authored
      Maintain consistency with the slapOS SR format.
      This SR can be hooked with other SR(ex:wendelin) and its better
      to follow one way of publishing result parameters
      
      [ kirr: This essentially changes publication format to JSON:
      
          $ xslapos proxy show --params
      
          # before
          slappart0: ipython_notebook (type default)
              url = https://[2001:67c:1254:e:49::952d]:8888
              monitor_url = https://[2001:67c:1254:e:49::952d]:9685
      
          # after
          slappart0: ipython_notebook (type default)
              _ = {"url": "https://[2001:67c:1254:e:49::952d]:8888", "monitor_url": "https://[2001:67c:1254:e:49::952d]:9685"}
      
        I'm not convinced we really need this, nor that the .serialized version is
        the most oftenly used one:
      
          slapos$ git grep 'slapos.cookbook:publish$' |wc -l
          59
          slapos$ git grep 'slapos.cookbook:publish.serialised$' |wc -l
          13
      
        but we can have it and see how it goes, reverting if needed ]
      
      /cc @jerome
      /proposed-for-review-on nexedi/slapos!33
      0fb4f687
    • Ayush Tiwari's avatar
      Jupyter: Set log-level alias to 'DEBUG' for ipython notebook service · 3bf1cce9
      Ayush Tiwari authored
      This helps in logging up the requests made by ipython_notebook service
      
      [ kirr: To be clear - until log-level is set to DEBUG, IPython notebook does
        not log HTTP requests, and since logging of HTTP requests is considered normal
        for most of our services (Zope, Apache, etc), it makes sense to enable such
        functionality for notebook too.
      
        There is not much additional noise produced by --log-level=DEBUG - in
        practice ipython only prints what config files it uses on startup, so this
        should be ok to go. ]
      
      /reviewed-by @kirr, @jerome  (on !33)
      3bf1cce9
    • Ayush Tiwari's avatar
      Jupyter: Hook ERP5 Kernel · 33954c66
      Ayush Tiwari authored
      ERP5 kernel basic info/workflow:
      
      1. User enters code on notebook cell and executes
      2. Code is sent to kernel via websockets
      3. Kernel sends request to ERP5
      4. Code is executed by ERP5 and the result is returned back via request.
      5. Result is received and rendered on the notebook frontend.
      6. Other message formats such as error and status are also conveyed by the Kernel.
      
      [ kirr: in IPython notebook speak kernel is something that allows IPython
        notebook server side to talk to execution backend. ERP5 kernel is a thing that
        allows ipython notbook to talk to ERP5 (with help on-ERP5-server special bt5
        installed which accepts and executes commands).
      
        The bt5 to handle notebook calls on ERP5 side - erp5-data-notebook - is
        proposed to be merged into erp5.git on nexedi/erp5!29 ]
      
      /initially-reviewed-by @kirr, @Tyagov  (in a lot of places, last time on nexedi/slapos!33)
      33954c66
    • Ayush Tiwari's avatar
      IPython Notebook: Explicitly add environment variable around wrapper and use... · 77ffa61e
      Ayush Tiwari authored
      IPython Notebook: Explicitly add environment variable around wrapper and use ipython directory inside instance in env
      
      [ kirr: By default IPython keeps configuration and other files location in
        ~/.ipython . What this patch does is organize explicit directory in instance
        tree to keep such files  ]
      
      /reviewed-by @kirr  (on nexedi/slapos!33)
      77ffa61e
    • Ayush Tiwari's avatar
      IPython Notebook: Add dynamic-template-base section for common jinja related... · 2ff9b784
      Ayush Tiwari authored
      IPython Notebook: Add dynamic-template-base section for common jinja related file section and extend them with this section
      2ff9b784
    • Ayush Tiwari's avatar
      IPython Notebook: Convert instance to jinja2 template · fbfaaef8
      Ayush Tiwari authored
      [ kirr: to-Jinja2 conversion is required because jinja is more suitable to
        describing instances compared to buildout, because jinja2 has e.g. control
        structures ]
      
      /reviewed-by @kirr  (on nexedi/slapos!33)
      fbfaaef8