gitlab: Make a plan to base instance layout on gitlab-omnibus and to interconnect all internal services via unix sockets

Upcoming changes will follow two points:

- we try to base our gitlab setup on how it is done in
  gitlab-omnibus[1] with the idea to ease tracking upstream changes to
  instance setup.

- we will interconnect all internal services via unix sockets only.

  The reason to do it is twofold:

    1. easier security: currently files on different slapos partitions
       are isolated from each other, but there is no "in-between-partitions"
       networking isolation - thus (potentially evil) programs can
       access internal services on other slapos partition.

       permissions to access unix sockets, on the other hand, are
       managed by filesystem-level permissions, and thus unix sockets in
       one partition will be, by default, isolated from programs on
       another partitions.

    2. It is well known that UNIX sockets are faster than TCP over
       loopback. For example for our std shuttles they have 2 times lower
       latency and ~ 2-3 times more throughput compared to TCP over loopback

    More details on 1 & 2 can be found e.g. here:

    nexedi/slapos!27
    https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/30

/cc @kazuhiko, @jerome

[1] https://gitlab.com/gitlab-org/omnibus-gitlab

Add stub instance configuration which just establishes a way to have
several software types(*), pass all needed info from software to
instance, organizes base directory and establishes log rotation base for
upcoming services.

Log rotation is done with the help of cron periodicallly calling
logrotate. The rotation is done in "copytruncate" mode - i.e. log file
is not moved away and signal sent for service to reopen it, but instead
log content is just copied to outside and there is no need for a service
to reopen it's log file.

The reason it is done this way, is that there is a chance of not
handling such "reopen-log-file" callbacks correctly on a service side,
and so the net is full of crashing reports, e.g. like this:

    http://serverfault.com/questions/627521/why-is-logrotate-causing-apache-to-seg-fault-each-time

That's why we take a safer approach instead, even if "copytruncate" mode
is risking to loose several log entries(**) on rotation.

NOTE services will organize log rotation with just

    [logrotate-entry-<service>]
    <= logrotate-entry
    log     = path/to/log/files/*.log

For this to work some "!py!" magic (our way to serialize object into
executable python and process it in buildout recipes) is used to process
section names.

The approach trick is also used for cron, e.g. logrotate registers to
cron this way:

    [cron-entry-logrotate]
    <= cron-entry
    time    = daily
    command = ${logrotate:wrapper}

NOTE2 instance md5 are not fixed yet - we'll fix them after applying all
    patches in gitlab series.

(*) for now there is only 1 - "gitlab", but we'll need to have "-export"
    and "-import" for resiliency in the future.

(**) ideally such things should be done with logfs - a filesystem
    specializeing in logging - for client services it will look like as
    they just continue to write to log file, and on log service side, the
    rotation can happen, all transparent to client service.

/cc @kazuhiko, @jerome

First step - build all needed software. We build:

- Git
- PostgreSQL 9.2
- Redis 2.8
- Nginx

- gitlab-shell
- gitlab-workhorse
- gitlab-ce 8.2 itself

and everything which is needed to build the above programs.

Git is needed because GitLab is a git-hosting service and uses git
underneath. PostgreSQL is used as DB by gitlab and Redis as a cache.

GitLab-shell is a small project to manage ssh access to the service
(we'll disable ssh though) and to perform all "change a repository"
operations.

GitLab-workhorse is a service which offloads long-running or slow
request from main GitLab service.

GitLab-ce is the main Ruby-on-Rails-based web application.

Ruby- and Go- based programs are built in a way similar to:

    - 31a45a94    (helloworld & helloweb: Ruby version), and
    - 24e82414    (helloworld & helloweb: Go version)

Version of all components, except Git, were picked the same, as used by
gitlab omnibus v8.2 .

/cc @kazuhiko, @jerome

Some dists like SLE_12 don't seem to have it.

../../stack/slapos.cfg is removed from component/*/buildout.cfg
because we normally don't specify it in component/
The OBS package will need to extend it.

Pin versions required for ipython==4.0.0 with ipykernel separated
from ipython eggs.
The split was in accordance to : https://blog.jupyter.org/2015/04/15/the-big-split/

/reviewed-by @kirr  (on nexedi/slapos!33)

ipython_notebook SR hooked with ERP5 kernel.
This kernel helps in interaction between erp5 and Jupyter frontend.
The patches have been cleaned up

Features:

- All the code execution is being done at erp5 side, Jupyter just acts as dumb client.
- Receives result as string and its mime_type and thanks to kernel, displays it accordingly.
- Interactions b/w erp5 and Jupyter frontend are based on HTTP requests.

Major changes:

- Addition of erp5 kernel
- Improvement in code according to guidelines(name, section name)
- Use jinja template as instance file and make it more dynamic
- Debugging added for ipython_notebook service.

Note: The certificate authentication changed has been reverted to the previous
one(done by creating wrapper around openssl command) for now.

/cc @Tyagov
/reviewed-by @kirr, @jerome  (on nexedi/slapos!33)

@jerome added --matplotlib=inline in 48eefab5 (ipython notebook) but it is
really neither needed:

   @jerome
   I remember adding this --matplotlib=inline line, but I am not sure it was
   ever needed. Using magic %matplotlib in notebook should be enough.

   @tiwariayush
   Yeah, for inline matplotlib in default python kernel, magics do there
   work(therefore neither pylab nor matplotlib alias are needed while starting the
   server), so I'd say leave this commit as it is and regarding version updation:
   a new patch making change wherever required.

nor supported:

   $ cat .slappart0_ipython_notebook.log
   [W 15:51:35.454 NotebookApp] Unrecognized alias: '--matplotlib=inline', it will probably have no effect.

Remove it.

P.S.

'--logfile' isn't available for ipython version 3.2.0 but we are not removing
it since we are planning to upgrade IPython to versions 4.x where it is supported.

Based on patch by @tiwariayush  (see nexedi/slapos!33)

Jupyter: Change section name to instance-jupyter so as not to raise conflict in case of multiple extends

/reviewed-by @kirr  (on nexedi/slapos!33)

Maintain consistency with the slapOS SR format.
This SR can be hooked with other SR(ex:wendelin) and its better
to follow one way of publishing result parameters

[ kirr: This essentially changes publication format to JSON:

    $ xslapos proxy show --params

    # before
    slappart0: ipython_notebook (type default)
        url = https://[2001:67c:1254:e:49::952d]:8888
        monitor_url = https://[2001:67c:1254:e:49::952d]:9685

    # after
    slappart0: ipython_notebook (type default)
        _ = {"url": "https://[2001:67c:1254:e:49::952d]:8888", "monitor_url": "https://[2001:67c:1254:e:49::952d]:9685"}

  I'm not convinced we really need this, nor that the .serialized version is
  the most oftenly used one:

    slapos$ git grep 'slapos.cookbook:publish$' |wc -l
    59
    slapos$ git grep 'slapos.cookbook:publish.serialised$' |wc -l
    13

  but we can have it and see how it goes, reverting if needed ]

/cc @jerome
/proposed-for-review-on nexedi/slapos!33

This helps in logging up the requests made by ipython_notebook service

[ kirr: To be clear - until log-level is set to DEBUG, IPython notebook does
  not log HTTP requests, and since logging of HTTP requests is considered normal
  for most of our services (Zope, Apache, etc), it makes sense to enable such
  functionality for notebook too.

  There is not much additional noise produced by --log-level=DEBUG - in
  practice ipython only prints what config files it uses on startup, so this
  should be ok to go. ]

/reviewed-by @kirr, @jerome  (on nexedi/slapos!33)

ERP5 kernel basic info/workflow:

1. User enters code on notebook cell and executes
2. Code is sent to kernel via websockets
3. Kernel sends request to ERP5
4. Code is executed by ERP5 and the result is returned back via request.
5. Result is received and rendered on the notebook frontend.
6. Other message formats such as error and status are also conveyed by the Kernel.

[ kirr: in IPython notebook speak kernel is something that allows IPython
  notebook server side to talk to execution backend. ERP5 kernel is a thing that
  allows ipython notbook to talk to ERP5 (with help on-ERP5-server special bt5
  installed which accepts and executes commands).

  The bt5 to handle notebook calls on ERP5 side - erp5-data-notebook - is
  proposed to be merged into erp5.git on erp5!29 ]

/initially-reviewed-by @kirr, @Tyagov  (in a lot of places, last time on !33)

IPython Notebook: Explicitly add environment variable around wrapper and use ipython directory inside instance in env

[ kirr: By default IPython keeps configuration and other files location in
  ~/.ipython . What this patch does is organize explicit directory in instance
  tree to keep such files  ]

/reviewed-by @kirr  (on !33)

IPython Notebook: Add dynamic-template-base section for common jinja related file section and extend them with this section

[ kirr: to-Jinja2 conversion is required because jinja is more suitable to
  describing instances compared to buildout, because jinja2 has e.g. control
  structures ]

/reviewed-by @kirr  (on nexedi/slapos!33)

Commit cee110b2 (IPython Notebook: Fixing coding crimes for section
names) changed IPython notebook section name to use '-' as word
delimiter but forgot to update users, and this way wendelin build
started to fail:

    INFO While:
    INFO   Installing.
    INFO   Getting section ipython_notebook.
    INFO Error: The referenced section, 'ipython_notebook', was not defined.

Fix it. (And I've made sure with whole-tree git grep that there is no
more ipython notebook users except wendelin in whole slapos.git so far)

/reported-by @Tyagov
/cc @tiwariayush
/reviewed-by TrustMe

This is required so that buildout does not fallback to installing
non-dev egg if version of wendelin.core from -dev differs from what has
been pinned.

For example the following

    [buildout]
    extends =
      https://lab.node.vifib.com/nexedi/slapos/raw/1.0.12/software/wendelin/software-dev.cfg

    # pin wendelin.core-dev to latest assumed-good revision with ZBlk1 support
    [wendelin.core-repository]
    revision = c507d9009f59fec2041bac9c31c5b08a48d3897d

will install wendelin.core-0.4.egg from pypi instead of installing
c507d9009f59fec2041bac9c31c5b08a48d3897d from repository, because that
latter revision says it is already version 0.5 and 1.0.12 wendelin SR
pins wendelin.core to 0.4 .

So unpin wendelin.core from versions and let software-dev.cfg work
always.

/cc @klaus
/reviewed-by @Tyagov  (on nexedi/slapos!36)

IPython Notebook: Add download-file-base section which would be extended by sections requiring common usage

/reviewed-by @kirr  (on nexedi/slapos!33)

/reviewed-by @kirr  (on nexedi/slapos!33)

Use spinal-case('-' separated) instead of snake_case('_' separated) in section names.

/reviewed-by @kirr  (on nexedi/slapos!33)

If restrict mode is set to true, nat interface (eth0) will be isolated, no network access, only host and guest forward rules will work throught that interface.
This option is true by default for kvm-cluster, and false for single and resilient kvm.

In case there are errors when creating cluster / setting up its
configuration files, currently we leave pgsql database left
half-installed and next time instantiation runs do not do anything,
because os.path.exists(pgdata) is already true.

I've personally hit this situation via providing ipv4 and ipv6
parameters as strings and the recipe wanted to do `ipv4.join(ipv6)` but this
works only for sets and raises for strings.

What is worse is that the above error becomes hidden in our default
setup, because webrunner tries to do instantiation _several_ times, and
on the second run instantiation succeeds, because pgdata directory
already exists and recipe thinks there is nothing to do _and_ webrunner
already removed instance.log from previous run.

So do not hide errors, and if we see there are problems, remove the
wholly created pgsql database directory.

/cc @kazuhiko, @jerome
/proposed-for-review on nexedi/slapos!29

If the list of families does not change, their ports must not change, and it's
wrong to get this by relying on CPython implementation details. Even if we
automated the update of frontends with new urls, this couldn't be done
atomically and we'd get random failures.

Currently, frontends are only updated manually so we also want to minimize
changes when families are added/renamed/removed. By sorting alphabetically,
we have something predictable. Of course, this does not cover cases like the
following one:
- before: A, B, C
- after: A, C
Even if we added a 'port-base' parameter for the balancer, the port would
change for one of the 2 families.

We have no need for the moment, but we could go further with an optional list
parameter to choose the order, and a special value to skip ports. Another
option is to use publish-early but it's more complicated to implement and
we lose everything when we reinstanciate.

The sort in haproxy.cfg.in is for the stats page.