Sidekiq[1] is used in GitLab as background jobs manager - i.e. if a
request handler needs to spawn some non-light job - it adds it to
sidekiq queue (in Redis) and relies on sidekiq service to later pick
this job up and execute it.

The service is setup with just to run bin/gitlab-sidekiq with
appropriate queues (extracted from omnibus-gitlab) and appropriate
settings to controlling GitLab's sidekiq Out-Of-Memory killer[2].

NOTE Unlike unicorn OOM killer, Sidekiq memory killer just makes sidekiq
    processes to be SIGKILL terminated and relies on managing service to
    restart it. In slapos we don't have mechanism to set autorestart=true,
    nor bang/watchdog currently work with slapproxy, so we setup to do
    such monitoring ourselves manually with here-introduced
    watcher-sigkill program.

NOTE2 sidekiq promise, because it is rake/gitlab based, is slow to
    load/run and thus is put into etc/promise.slow/

[1] http://sidekiq.org/
[2] https://gitlab.com/gitlab-org/gitlab-ce/blob/1322bd78/doc/operations/sidekiq_memory_killer.md

/cc @kazuhiko, @jerome

As was described in the previous patch, we need $HOME to be propagated
by this programs so that git can find partition's .gitconfig.

Specifically we need the following patches to be present in our build:

    https://gitlab.com/gitlab-org/gitlab-shell/commit/9e087f64
    https://gitlab.com/gitlab-org/gitlab-workhorse/commit/b5f1b803

They both have been applied upstream very close to revisions we
previously had in software.cfg, so we only need to update the revisions
to get them.

/cc @kazuhiko, @jerome

Convert gitconfig template to jinja2 (reusing already-there
`email_display_name` and `email_from` parameters for commits generated by
gitlab).

System-level git config from gitlab-omnibus is also imported to this
file (on slapos we cannot tweak system-level git config -
software/.../parts/git/... is read-only for programs in instance
partitions - so we move all gitlab's system-wide git settings to this
"user-level" gitconfig.

System gitconfig in omnibus is defined here:

    https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda093/files/gitlab-cookbooks/gitlab/attributes/default.rb#L23

so it is

    pack.threads = 1    and
    receive.fsckObjects = true

which makes sense to not waste a lot of memory when packing and not to
allow corrupt objects to enter to system by evil users intentionally.

To make the file foundable by git - we put it into partition root
directory and set $HOME to point to partition root when running
appropriate programs / services.

NOTE we'll need to upgrade gitlab-shell and gitlab-workhorse to
    propagate $HOME for this setting to actually have effect.
    See the next patch.

/cc @kazuhiko, @jerome

Like with Rails configuration files, this is pristine import of template
gitconfig from omnibus GitLab from

    https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default

This is only a "user" part of git configuration. System-wide
configuration is generated dynamically:

    https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda093/files/gitlab-cookbooks/gitlab/attributes/default.rb#L23

and we'll import it by hand in the follow-up patches.

/cc @kazuhiko, @jerome

Gitlab-workhorse[1] is a service which offloads Ruby-on-Rails based
GitLab from long-running and slow requests. It is written in Go.

Now as we have unicorn service set up, we can setup gitlab-workhorse
service (which uses unicorn as authentication backend).

Gitlab-workhorse setup is easy - it is just one program and several
command line options to point to unicorn socket and to configure on
which unix socket gitlab-workhorse will listen itself.

NOTE we have to care that git and ruby to be on PATH when running
    gitlab-workhorse - because on e.g. git push'ing workhorse will run `git
    receive-pack` and a hook will be called which calls gitlab-shell,
    which is written in ruby.

NOTE2 promise to check whether gitlab-workhorse is alive is to ping it
    via URL to non-existent endpoint and check for proper 403 HTTP code
    returned.

[1] https://gitlab.com/gitlab-org/gitlab-workhorse

/cc @kazuhiko, @jerome

gitlab: Compile assets on instantiation and make sure DB is properly setup/migrated before unicorn runs

There are several actions that needs to be done on gitlab instance
upgrade:

    - we have to (re-)compile assets
    - we have to migrate DB

and also before the first run

    - we have to initialize DB

We can compile assets as part of instantiation process, but regarding
DB migration / setup - it is not currently possible to do that as part
of instantiation - for that operations we need PG & Redis to be already
running, but the first time slapos instantiates an SR it first prepares
all services, and only after instantiation is done, starts them all. There is
currently no way to hook into starting process, and run some scripts
after one service is started but before another service startup...

So the solution is: to perform such actions in delayed mode as part of
application - unicorn service - startup: it makes sure PG is running and
initializes it and does other actions which needs to be done to migrate
the DB. Only if/after they succeed the main application is started.

NOTE the comment about unicorn/gitlab startup slowness from the previous
    patch still holds true - so in order to get "all ok" after
    instantiation, it is required to perform the instantiation several
    times, because unicorn promise initially fails.

/cc @kazuhiko, @jerome

Now that all gitlab Rails application configuration files are ready, we
can setup unicorn service to start it.

NOTE there is a promise to check unicorn by url which works, but there
    are also rake tasks to check gitlab itself, e.g. like

        gitlab:app:check
        gitlab:gitlab_shell:check
        gitlab:repo:check

    Unfortunately this tasks are slow to run (and gitlab:repo:check is
    very slow to run). That's why we do not put them into etc/promise/ -
    if we do - slapos reports promises time outs.

    What we do is we put them into etc/promise.slow/ so we have those
    scripts ready, but currently no one automatically checks them.

    Again, the promise to check unicorn just by accessing it by URL is
    there and is checked automatically out of the box.

NOTE2 GitLab is very slow to load. That's why it can take some time
    after unicorn starts that it's promise start to report ok. This can
    show itself as temporary instantiation errors which say promise such and
    such failed.

NOTE3 Unicorn start, but so far we did not cared to setup GitLab DB
    schema on instantiation. That's why unicorn remains not very usable
    and a lot of requests fail. We'll teach instance to setup DB and
    perform all other needed settings in the next patch.

/cc @kazuhiko, @jerome

Like with [promise-wrapper] a recipe could do

    [promise-<service>]
    <= promise-byurl
    url     = ...

and a script to check such ur will be generated and automatically put
into etc/promise/<service>.

/cc @kazuhiko, @jerome

Automatically configure unicorn to spawn as much worker processes as
there are CPUs on the system by default.

GitLab omnibus pre-hardcodes this value default to 2 (which we copied)
and then also tweaks it this way in active code

    https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/attributes/default.rb#L230

which we also do here.

/cc @kazuhiko, @jerome

Explicitly point gitlab-shell to location where we keep secrets.

We already pointeg gitlab to that place and now we do that for
gitlab-shell so those 2 peieces can connect to each other ok.

Regarding the setting itself - there is no such block in omnibus-gitlab,
but it is present in gitlab-shell configuration example:

    https://gitlab.com/gitlab-org/gitlab-shell/blob/82b3a4e8/config.yml.example#L35

/cc @kazuhiko, @jerome

Convert gitlab-shell configuration file to slapos:

    - convert to jinja2,
    - connect gitlab-shell to unicorn & redis unix sockets

NOTE

    - http_settings are left to be default (empty) ones - as that works ok.
    - `auth_file` is still configured to point to wont-be-used sshkeys
      file, as without it gitlab-shell check will fail.
    - support for audit_usernames and git_annex is disabled and
      remains not configurable.

/cc @kazuhiko, @jerome

Unicorn is a forking server with the idea that master process preloads
heavy Ruby-on-Rails application, and then to handle new request a worker
process is forked with application already loaded in its memory (and
modification being tracked by OS via copy-on-write).

From this point of view the only reasonable value for preload_app is
always "true" and omnibus-gitlab does this:

    https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/definitions/unicorn_service.rb#L65

Then unicorn documentation shows what code has to be there in pre-/post-
forking event:

    http://bogomips.org/unicorn.git/tree/examples/unicorn.conf.rb?id=3312aca8#n57

GitLab uses only part of it that "allows a new master process to
incrementally phase out the old master process with SIGTTOU to avoid a
thundering herd":

    https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8eda0933/files/gitlab-cookbooks/gitlab/definitions/unicorn_service.rb#L69
    http://bogomips.org/unicorn.git/tree/examples/unicorn.conf.rb?id=3312aca8#n75

but strangely does not use code parts that are "highly recommended" or
"require" for "Rails + "preload_app true"" case.

For the reference I've added such codes, but kept them being commented
out.

/cc @kazuhiko, @jerome

Convert unicorn parameters to slapos and configure it to listen on unix
socket only.

( Omnibus configures unicorn to listen on unix socket and
  loopback TCP, mainly because gitlab-shell could not connect to unicorn
  via unix socket until recently:

      https://gitlab.com/gitlab-org/gitlab-shell/commit/184385ac

  But as it can now, there is no point to keep on TCP port open )

To be able to do such configuration we add stub to unicorn service
section (to create needed directories where to keep the socket).

There will be follow-up patch which configures unicorn pre/post-forking
actions, which is not trivial and thus better be done on its own.

/cc @kazuhiko, @jerome

Convert the rest of this configuration file to slapos.

It is straightforward conversion of parameters except:

    - access-via-ssh is disabled (gitlab slapos version does not support
      ssh access and supports HTTP(S) only by design on purpose)

    - we do not support restricting possible projects visibility via
      instance parameter (very low chance this will be needed in
      practice)

    - default issue-closing pattern is just ok for now and not
      configurable

    - support for builds, build artifacts & CI is disabled (we do not
      support CI (yet ?))

    - some internal defaults are just ok (e.g. where to organize
      directory for keeping repositories archives for downloads)

    - reply-by-email is not supported (yet ?)

    - we do not support LFS (yet ?) - just plain git hosting is ok for now.

    - Gravatar defaults are ok for now and not configurable.

    - Support for LDAP is disabled

    - Support for Kerberos is disabled

    - Support for OmniAuth is disabled

    - Satellites path is just /dev/null as we start from version where
      satellites are already non-existent.

    - Uploading backups to somewhere via GitLab's builtin mechanism is
      not supported - we'll use SlapOS native backup and resiliency for
      this.

    - Support for Google analytics is disabled.

    - Support for Piwik is disabled.

    - we are ok (for now) with default rack-attack git settings

/cc @kazuhiko, @jerome

This user will need to be specified several times in configuration
files, as by default gitlab uses 'git' user and does "sudo" to it if it
is not current.

We will use {{ backend_info.user }} in the upcoming patches.

/cc @kazuhiko, @jerome

GitLab has a notion of "external URL" - the canonical "frontend" URL the
server is reachable through: this URL is used as prefix to show
e.g. git-clone URL for repositories, etc, even if a server can be
reachable via several frontends.

Add external_url handling to slapos instance.

NOTE whether to use https or not is also defined by external_url, in
particular by external_url scheme.

/cc @kazuhiko, @jerome

Convert to slapos SMTP settings for gitlab:

    - convert to jinja2
    - remove support for gitlab CI (we do not support it (yet ?))
    - add handling of `smtp_enable` parameter directly to that file
      ( omnibus handles this parameter externally and just removes
        smtp_settings.rb if it is true )

NOTE smtp_settings.rb contains SMTP password, so it is mode is set to 0600.

/cc @kazuhiko, @jerome

Just another 2 simple parameters (attack detection tunables) conversion
to jinja2/slapos.

/cc @kazuhiko, @jerome

Just convert 2 parameters used in that file to jinja syntax and add
those parameters (unicorn OOM killer tunables) to gitlab-parameters.cfg

/cc @kazuhiko, @jerome

A simple change just to point resque to redis unix socket.

/cc @kazuhiko, @jerome

We tweak database.yml to point to our postgresql unix socket; set
adapter to hardcoded postgresql, encoding to unicode and omit collation
(which according to omnibus-gitlab is used for mysql only).

The only instance parameter imported from omnibus is `db_pool` - how
many connection to a DB to keep open in a RoR thread/process.

XXX we use db's superuser as a user to connect. Is it ok to do even if
    the whole DB is used only for gitlab? (I think it is ok for the
    first iteration, but we'll probably need to refine this later)

/cc @kazuhiko, @jerome

Introduce a library of Jinja2 macros that will be handy to use in
templates. For now we add only 2 macros:

    cfg(name)   - to get instance configuration parameter `name`,   and
    cfg_bool    - to get truth value of ----//----

The reason we introduce cfg() is that we will need to use a lot of
parameters in many places and it is much more handy to write, e.g.

    cfg('email_enabled')

compared to

    instance_parameter:configuration.email_enabled

/cc @kazuhiko, @jerome

We will be using a several dozens of parameters to control gitlab
instance. It makes sense not to deviate in such parameters namings and
defaults from omnibus version.

Thus for such parameters - for clarity - we organize a separate file
where we will be keeping them - gitlab-parameters.cfg.

In this patch series all used parameters will be "imported" from
omnibus-gitlab 8.2.3+ce.0-0-g8eda093.

NOTE it is maybe better to try to autogenerate that file from upstream
    omnibus parameters definitions. If time will tell it becomes hard to
    maintain our copy - we'll consider going that way.

/cc @kazuhiko, @jerome

- Download them on SR build and pass info to instance
- Instance prepares to process them as jinja2 templates
- Instance hooks the files into configuration location as appropriate

Every file so far is renamed *.erb -> *.in and a header added showing
that this file is autogenerated with links about what was the base
gitlab and/or omnibus version and omnibus reference revision this
template was last updated for.

So far all result configuration files are invalid - because ERB syntax
is there. We will convert the configuration files to proper jinja2
syntax and to using slapos parameters incrementally in the upcoming
patches.

NOTE (again): md5 sums are not yet fixed - we will fix them in the end
    of gitlab patches series after applying all tweaking changes.

/cc @kazuhiko, @jerome

Pristine import of template configuration files from omnibus GitLab
package. All files were imported as-is in their ERB form and filenames
from omnibus-gitlab 8.2.3+ce.0-0-g8eda093 from here:

    https://gitlab.com/gitlab-org/omnibus-gitlab/tree/8eda093/files/gitlab-cookbooks/gitlab/templates/default

We will convert the templates to jinja2 and adjust them to slapos
version in the following patches.

Scheme for synchronizing with future upstream changes is envisioned as this:

    - checkout latest commit which updated pristine erb files
    - copy updated files from omnibus-gitlab, and commit the updates
    - checkout slapos master
    - merge commit that updated erb

That should reasonably work with not too-many conflicts and even those
should be not hard to resolve (with `git mergetool` e.g. in kdiff3)

/cc @kazuhiko, @jerome

Organize per-instance place for gitlab configuration and work directory.

Unfortunately as GitLab is Ruby-on-Rails application, it is not possible
to keep its code in one place and have multiple separate configuration
sets in different places and start that code for a configuration set -
GitLab and Rails insist to get configuration from relative to source
code tree.

GitLab omnibus "solves" this by having only one configuration set and
having symlinks from code to that only configiration set. In slapos we
can potentially have several instances for one software and thus we
cannot do that.

With such limitations a proper solution would be to bind-mount software
code into instance filesystem namespace close to configuration - that
way the code will be only one and will find proper per-instance config.
Currently we do not have namespaces available on slapos unfortunately,
thus something else is needed.

The workaround I decided to do is this: to clone cloned gitlab
repository from software/ space to instance/ space and adjust it in
instance space. This has the following drawbacks:

    - code is duplicated
    - code becomes read-write, instead of being read-only

but imho it is the most practical thing to do. Another solution could be
to patch GitLab / Rails to remove "config lives in code" assumption, but
the number of places where this needs to be done is really many.

NOTE gems which gitlab uses and which were installed during software
    compilation are not duplicated - they are reused via bundler - via
    pointing BUNDLE_GEMFILE to original location in software.

NOTE2 For instance tasks and also for maintanace convenience we establish
    <instance>/bin/gitlab-* programs, e.g. gitlab-rake, which e.g. for
    gitlab-rake will run rake with correctly loaded gitlab environment -
    like in gitlab-omnibus.

/cc @kazuhiko, @jerome, @jp

Organize internal Redis service, like with PostgreSQL in the previous
patch, with the help of slapos.cookbook:redis.server recipe.

Like with postgresql, and as we planned, redis listens only on
internal-to-partition unix socket.

The recipe establishes both service and promise to check it is alive;
we only need to setup log rotation manually.

/cc @kazuhiko, @jerome

Organize internal PostgreSQL database which will be used as DB for
Roby-on-Rails GitLab and listens only on unix socket (for security and
performance reasons - see earlier intro patch).

To do it we use slapos.cookbook:postgres recipe, with disabling
"listen-to-network" via passing empty sets to ipv4 and ipv6 recipe
arguments.

The promise to check whether DB is alive is just `psql -c '\q'` which
will error if failing to connect to DB, but exit silently if connected ok.

Explicit log rotation is not needed - as postgresql logs to
stdout/stderr - not to a file - logs are handled by slapos - put into
.slappartX_postgresql.log and automatically rotated there.

XXX omnibus-gitlab tunes postgresql with shared_buffers and other
parameters, most likely for performance reasons - see e.g.

    https://gitlab.com/gitlab-org/omnibus-gitlab/blob/8-2-stable/files/gitlab-cookbooks/gitlab/templates/default/postgresql.conf.erb#L113

I decided not to fine-tune postgresql for now, and get on-field feedback
first, and then, if needed, we can tune.

/cc @kazuhiko, @jerome

A recipe could do

    [promise-<service>]
    <= promise-wrapper
    command-line = ...

and the wrapper will be put automatiaclly into etc/promise/<service>.

( for this to happen !py! magic is used again, like we did for logrotate
  and cron entries before )

/cc @kazuhiko, @jerome

gitlab: Make a plan to base instance layout on gitlab-omnibus and to interconnect all internal services via unix sockets

Upcoming changes will follow two points:

- we try to base our gitlab setup on how it is done in
  gitlab-omnibus[1] with the idea to ease tracking upstream changes to
  instance setup.

- we will interconnect all internal services via unix sockets only.

  The reason to do it is twofold:

    1. easier security: currently files on different slapos partitions
       are isolated from each other, but there is no "in-between-partitions"
       networking isolation - thus (potentially evil) programs can
       access internal services on other slapos partition.

       permissions to access unix sockets, on the other hand, are
       managed by filesystem-level permissions, and thus unix sockets in
       one partition will be, by default, isolated from programs on
       another partitions.

    2. It is well known that UNIX sockets are faster than TCP over
       loopback. For example for our std shuttles they have 2 times lower
       latency and ~ 2-3 times more throughput compared to TCP over loopback

    More details on 1 & 2 can be found e.g. here:

    nexedi/slapos!27
    https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/30

/cc @kazuhiko, @jerome

[1] https://gitlab.com/gitlab-org/omnibus-gitlab

Add stub instance configuration which just establishes a way to have
several software types(*), pass all needed info from software to
instance, organizes base directory and establishes log rotation base for
upcoming services.

Log rotation is done with the help of cron periodicallly calling
logrotate. The rotation is done in "copytruncate" mode - i.e. log file
is not moved away and signal sent for service to reopen it, but instead
log content is just copied to outside and there is no need for a service
to reopen it's log file.

The reason it is done this way, is that there is a chance of not
handling such "reopen-log-file" callbacks correctly on a service side,
and so the net is full of crashing reports, e.g. like this:

    http://serverfault.com/questions/627521/why-is-logrotate-causing-apache-to-seg-fault-each-time

That's why we take a safer approach instead, even if "copytruncate" mode
is risking to loose several log entries(**) on rotation.

NOTE services will organize log rotation with just

    [logrotate-entry-<service>]
    <= logrotate-entry
    log     = path/to/log/files/*.log

For this to work some "!py!" magic (our way to serialize object into
executable python and process it in buildout recipes) is used to process
section names.

The approach trick is also used for cron, e.g. logrotate registers to
cron this way:

    [cron-entry-logrotate]
    <= cron-entry
    time    = daily
    command = ${logrotate:wrapper}

NOTE2 instance md5 are not fixed yet - we'll fix them after applying all
    patches in gitlab series.

(*) for now there is only 1 - "gitlab", but we'll need to have "-export"
    and "-import" for resiliency in the future.

(**) ideally such things should be done with logfs - a filesystem
    specializeing in logging - for client services it will look like as
    they just continue to write to log file, and on log service side, the
    rotation can happen, all transparent to client service.

/cc @kazuhiko, @jerome

First step - build all needed software. We build:

- Git
- PostgreSQL 9.2
- Redis 2.8
- Nginx

- gitlab-shell
- gitlab-workhorse
- gitlab-ce 8.2 itself

and everything which is needed to build the above programs.

Git is needed because GitLab is a git-hosting service and uses git
underneath. PostgreSQL is used as DB by gitlab and Redis as a cache.

GitLab-shell is a small project to manage ssh access to the service
(we'll disable ssh though) and to perform all "change a repository"
operations.

GitLab-workhorse is a service which offloads long-running or slow
request from main GitLab service.

GitLab-ce is the main Ruby-on-Rails-based web application.

Ruby- and Go- based programs are built in a way similar to:

    - 31a45a94    (helloworld & helloweb: Ruby version), and
    - 24e82414    (helloworld & helloweb: Go version)

Version of all components, except Git, were picked the same, as used by
gitlab omnibus v8.2 .

/cc @kazuhiko, @jerome

Some dists like SLE_12 don't seem to have it.

../../stack/slapos.cfg is removed from component/*/buildout.cfg
because we normally don't specify it in component/
The OBS package will need to extend it.

Pin versions required for ipython==4.0.0 with ipykernel separated
from ipython eggs.
The split was in accordance to : https://blog.jupyter.org/2015/04/15/the-big-split/

/reviewed-by @kirr  (on nexedi/slapos!33)