Skip to content

slapos
slapos
Commit 9e01d769 authored by iv's avatar iv Committed by Kirill Smelkov
3 Browse files
Options

gitlab: Upgrade to 8.8.9 + start automation of gitlab-backup 

# Upgrade

    * GitLab Software + patches ported to GitLab 8.8.9;
    * Configs synced with upstream and SlapOS-ified;
    * Fix promises due to curl upgrade

@jerome @kazuhiko @kirr: if you have any comments, I would be glad to read them. There is a [demo instance](https://softinst69126.host.vifib.net) you can try.

Check on the demo instance for the Gitlab configuration:

- [x] migrations are all `up`, when running: `gitlab-rake db:migrate:status`
- [x] promises don't fail (including slow ones)
- [x] web UI seems fine
- [x] git clone/fetch/push works well
- [x] email notifications are sent

and Nexedi edition patches:

- [x] raw blobs download
- [x] private ones with gitlab-ci token `curl -v https://user:token@testinstance/kirr/test/raw/master/hello.txt`
- [x] applying patches in merge-request
- [x] merge as topic in merge-request
- [x] "Nexedi edition" being present on front page and on page footers with ICP
- [x] clone protocol is only HTTPS (not SSH)

No changes are made in gitlab-workhorse and gitlab-shell upstream this time, but Nexedi patches and a cherry-picked commit are applied in a merge request on gitlab-ce.

Note: in order to build the gitlab SR, you need to change gitlab-ce repository to my own, as it is not merged yet:
```
--- a/software/gitlab/software.cfg
+++ b/software/gitlab/software.cfg
@@ -115,7 +115,7 @@ git-executable = ${git:location}/bin/git
 [gitlab-repository]
 <= git-repository
 #repository = https://gitlab.com/gitlab-org/gitlab-ce.git
-repository = https://lab.nexedi.com/nexedi/gitlab-ce.git
+repository = https://lab.nexedi.com/iv/gitlab-ce.git
```

# Gitlab backup (pull automation)

The idea is to regularly do a gitlab-backup pull for now, so that later, we can add some code to configure the other gitlab instance (not in production) to pull the created backup repo.

/reviewed-on nexedi/slapos!124
parents ee91827b ae7b4120
Hide whitespace changes
Inline Side-by-side
Showing with 202 additions and 29 deletions
slapos/recipe/wrapper.py
View file @ 9e01d769
......@@ -37,6 +37,7 @@ class Recipe(GenericBaseRecipe):
wait_files = self.options.get('wait-for-files')
environment = self.options.get('environment')
parameters_extra = self.options.get('parameters-extra')
pidfile = self.options.get('pidfile')
if not wait_files and not environment:
# Create a simple wrapper as shell script
......@@ -45,6 +46,7 @@ class Recipe(GenericBaseRecipe):
command=command_line[0],
parameters=command_line[1:],
parameters_extra=parameters_extra,
pidfile=pidfile,
)]
# More complex needs: create a Python script as wrapper
......@@ -72,5 +74,7 @@ class Recipe(GenericBaseRecipe):
name=wrapper_path,
command=python_script,
parameters=[],
parameters_extra=parameters_extra) ]
parameters_extra=parameters_extra,
pidfile=pidfile,
)]
software/gitlab/gitlab-export.in 0 → 100644
View file @ 9e01d769
#!{{ bash.location }}/bin/bash -e
# export gitlab site via `gitlab-backup pull` to a "for-export" git-backup repository
# gitlab-export <exportto-repo>
#
# NOTE gitlab-backup, gitlab-rails, ... all have to be on $PATH.
# which site to export is determined by which gitlab-rails is on $PATH.
if [ "$#" -ne 1 ]; then
echo "Usage: gitlab-export <exportto-repo>" 1>&2
exit 1
fi
exportto_repo="$1"
# create / setup export repository if it does not exist yet
mkdir -p "$exportto_repo"
cd "$exportto_repo"
# verify we are in a git repository
if ! git rev-parse --is-inside-git-dir ; then
git init --bare
fi
exec gitlab-backup pull
software/gitlab/gitlab-parameters.cfg
View file @ 9e01d769
......@@ -6,7 +6,7 @@
#
# TODO better autogenerate from ^^^ (?)
#
# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
[gitlab-parameters]
configuration.external_url = http://lab.example.com
......
software/gitlab/instance-gitlab-export.cfg.in 0 → 100644
View file @ 9e01d769
# GitLab instance + site export support
[buildout]
extends = {{ instance_gitlab_cfg }}
# TODO + ${pbsready-export:output}
parts +=
cron-entry-gitlab-backup
# -export specific instance parameters
[instance-parameter]
# cron frequency for gitlab backup (default: every 4h)
configuration.backup_frequency = 0 */4 * * *
[gitlab-backup-directory]
recipe = slapos.cookbook:mkdirectory
srv = ${buildout:directory}/srv
backup = ${:srv}/backup
backup-gitlab.git = ${:backup}/backup-gitlab.git
var = ${buildout:directory}/var
pid = ${:var}/pid
# instance exporter script
[exporter]
recipe = slapos.cookbook:wrapper
wrapper-path = ${buildout:directory}/bin/gitlab-exporter
command-line = {{ xnice_repository_location }}/bin/xnice {{ gitlab_export }} ${gitlab-backup-directory:backup-gitlab.git}
pidfile = ${gitlab-backup-directory:pid}/gitlab-exporter.pid
environment =
# XXX: `/usr/bin` has to be in the PATH environment variable to be able to use
# `which` command in gitlab-backup, `chrt` in xnice, ...
PATH=/usr/bin:${buildout:directory}/bin:{{ coreutils_location }}/bin:{{ grep_location }}/bin:{{ tar_location }}/bin:{{ gzip_location }}/bin:{{ gopath_bin }}:{{ git_location }}/bin
[cron-entry-gitlab-backup]
<= cron-entry
# run backup script on a regular basis (given as instance parameter)
frequency = ${instance-parameter:configuration.backup_frequency}
command = ${exporter:wrapper-path}
software/gitlab/instance-gitlab.cfg.in
View file @ 9e01d769
......@@ -537,7 +537,7 @@ depend =
[promise-gitlab-workhorse]
<= promise-byurl
url = --unix-socket ${gitlab-workhorse:socket} http:/static.css
url = --unix-socket ${gitlab-workhorse:socket} http://localhost/static.css
# gitlab-workhorse logs to stdout/stderr - logs are handled by slapos not us
......@@ -579,7 +579,7 @@ depend =
[promise-unicorn]
<= promise-byurl
url = --unix-socket ${unicorn:socket} http:/
url = --unix-socket ${unicorn:socket} http://localhost/
[promise-rakebase]
recipe = slapos.cookbook:wrapper
......@@ -627,7 +627,7 @@ log = ${sidekiq-dir:log}
# NOTE see queue list here:
# https://gitlab.com/gitlab-org/gitlab-ce/blob/master/Procfile
# https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb
# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
[service-sidekiq]
recipe = slapos.cookbook:wrapper
wrapper-path = ${directory:service}/sidekiq
......
software/gitlab/instance.cfg.in
View file @ 9e01d769
......@@ -10,15 +10,17 @@ offline = true
[switch-softwaretype]
recipe = slapos.cookbook:softwaretype
default = $${instance-gitlab.cfg:rendered}
# TODO -export, -import, -pull-backup
gitlab = $${instance-gitlab.cfg:rendered}
gitlab-export = $${instance-gitlab-export.cfg:rendered}
default = $${:gitlab}
# TODO -import, -pull-backup
[instance-gitlab.cfg]
# macro: render instance-*.cfg from instance-*.cfg.in
[instance-cfg]
recipe = slapos.recipe.template:jinja2
mode = 0644
template= ${instance-gitlab.cfg.in:target}
rendered= $${buildout:directory}/instance-gitlab.cfg
rendered= $${buildout:directory}/$${:_buildout_section_name_}
context =
import os os
import pwd pwd
......@@ -33,12 +35,16 @@ context =
raw bash_bin ${bash:location}/bin/bash
raw bzip2_location ${bzip2:location}
raw bundler_4gitlab ${bundler-4gitlab:bundle}
raw coreutils_location ${coreutils:location}
raw curl_bin ${curl:location}/bin/curl
raw dcron_bin ${dcron-output:crond}
raw git ${git:location}/bin/git
raw git_location ${git:location}
raw gitlab_export ${gitlab-export:rendered}
raw gitlab_workhorse ${gitlab-workhorse:location}/gitlab-workhorse
raw gopath_bin ${gopath:bin}
raw gunzip_bin ${gzip:location}/bin/gunzip
raw grep_location ${grep:location}
raw gzip_bin ${gzip:location}/bin/gzip
raw gzip_location ${gzip:location}
raw logrotate_bin ${logrotate:location}/usr/sbin/logrotate
......@@ -48,7 +54,9 @@ context =
raw postgresql_location ${postgresql92:location}
raw redis_binprefix ${redis28:location}/bin
raw ruby_location ${bundler-4gitlab:ruby-location}
raw tar_location ${tar:location}
raw watcher_sigkill ${watcher-sigkill:rendered}
raw xnice_repository_location ${xnice-repository:location}
# config files
raw database_yml_in ${database.yml.in:target}
......@@ -64,3 +72,16 @@ context =
raw resque_yml_in ${resque.yml.in:target}
raw smtp_settings_rb_in ${smtp_settings.rb.in:target}
raw unicorn_rb_in ${unicorn.rb.in:target}
$${:context-extra}
context-extra =
[instance-gitlab.cfg]
<= instance-cfg
template= ${instance-gitlab.cfg.in:target}
[instance-gitlab-export.cfg]
<= instance-cfg
template= ${instance-gitlab-export.cfg.in:target}
context-extra =
raw instance_gitlab_cfg $${instance-gitlab.cfg:rendered}
software/gitlab/software.cfg
View file @ 9e01d769
......@@ -14,9 +14,12 @@ extends =
../../component/nginx/buildout.cfg
# for instance
../../component/coreutils/buildout.cfg
../../component/bash/buildout.cfg
../../component/grep/buildout.cfg
../../component/bzip2/buildout.cfg
../../component/curl/buildout.cfg
../../component/tar/buildout.cfg
../../component/gzip/buildout.cfg
../../component/dcron/buildout.cfg
../../component/logrotate/buildout.cfg
......@@ -36,20 +39,29 @@ parts =
gitlab-shell/vendor
gitlab/vendor/bundle
gitlab-workhorse
git-backup
# for instance
instance.cfg
# TODO stop using slapos.cookbook-develop once 571d6514 gets into released
# slapos.cookbook egg
slapos-cookbook-develop
slapos-cookbook
eggs
bash
curl
watcher-sigkill
gitlab-export
gzip
dcron-output
logrotate
[slapos.cookbook-repository]
revision = 571d6514f7290e8faa9439c4b86aa2f6c87df261
############################
# Software compilation #
......@@ -113,15 +125,15 @@ git-executable = ${git:location}/bin/git
<= git-repository
#repository = https://gitlab.com/gitlab-org/gitlab-ce.git
repository = https://lab.nexedi.com/nexedi/gitlab-ce.git
# 8.7.X + NXD patches:
revision = v8.7.9-10-g7728df66b90483dac467df95948a532783e782c8
# 8.8.X + NXD patches:
revision = v8.8.9-10-g967afbdc3a2d2bcc58ed31ca50d5293fa2a2e9ed
location = ${buildout:parts-directory}/gitlab
[gitlab-shell-repository]
<= git-repository
#repository = https://gitlab.com/gitlab-org/gitlab-shell.git
repository = https://lab.nexedi.com/nexedi/gitlab-shell.git
# gitlab 8.7 wants gitlab-shell 2.7.2
# gitlab 8.8 wants gitlab-shell 2.7.2
# 2.7.2 + NXD patches
revision = v2.6.10-50-gfbca95be784816349abc5930324659151eca50d1
location = ${buildout:parts-directory}/gitlab-shell
......@@ -130,7 +142,7 @@ location = ${buildout:parts-directory}/gitlab-shell
<= git-repository
#repository = https://gitlab.com/gitlab-org/gitlab-workhorse.git
repository = https://lab.nexedi.com/nexedi/gitlab-workhorse.git
# gitlab 8.7 wants gitlab-workhorse 0.7.1
# gitlab 8.8 wants gitlab-workhorse 0.7.1
# 0.7.1 + NXD patches
revision = v0.7.1-5-gd23a3247829fc3200e3dc784dcd57b5a0febac48
location = ${buildout:parts-directory}/gitlab-workhorse
......@@ -151,6 +163,52 @@ make-binary =
make-targets= cd ${:path} &&
${:bundle} install --deployment --without development test mysql kerberos
# directories and repositories required by gitlab-backup
[gopath]
directory = ${buildout:directory}/go.work
src = ${:directory}/src
bin = ${:directory}/bin
[go-git-repository]
<= git-repository
repository = https://${:go.importpath}.git
location = ${gopath:src}/${:go.importpath}
[git2go-repository]
<= go-git-repository
go.importpath = github.com/libgit2/git2go
# branch 'next' is required by git-backup
revision = next-g53594d7581617dbae7bb5960b4ac5f0ff513c184
[git-backup-repository]
<= go-git-repository
go.importpath = lab.nexedi.com/kirr/git-backup
revision = 3ba6cf73ba224c40f67f1fb87c855b915eb91f58
[git-backup]
# install git2go, git-backup, gitlab-backup in <gopath>/bin
recipe = slapos.recipe.cmmi
path = ${git-backup-repository:location}
configure-command = :
make-binary =
make-targets= cd ${git2go-repository:location}
&& git submodule update --init
&& make install
&& install -d ${gopath:bin}
&& go install ${git-backup-repository:go.importpath}
&& cp -a ${git-backup-repository:location}/contrib/gitlab-backup ${gopath:bin}
environment =
PATH=${cmake:location}/bin:${pkgconfig:location}/bin:${git:location}/bin:${golang16:location}/bin:${buildout:bin-directory}:%(PATH)s
GOPATH=${gopath:directory}
[xnice-repository]
# to get kirr's misc repo containing xnice script for executing processes
# with lower priority (used for backup script inside the cron)
<= git-repository
repository = https://lab.nexedi.com/kirr/misc.git
revision = 4073572ea700bf1b115f3a135aebebe5b3b824e4
location = ${buildout:parts-directory}/misc
# build needed-by-gitlab-shell gems via bundler
# ( there is not vendor/ dir in gitlab-shell, so to avoid having buildout error
......@@ -194,17 +252,26 @@ eggs =
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/instance.cfg.in
output = ${buildout:directory}/instance.cfg
md5sum = ef85f02c4f6070c586d773b859a2f4e2
md5sum = b99a99b161c0b292845002fc3fee50cd
[watcher-sigkill]
# macro: download a shell script and put it rendered into <software>/bin/
[binsh]
recipe = slapos.recipe.template:jinja2
template= ${:_profile_base_location_}/${:_buildout_section_name_}.in
rendered= ${buildout:bin-directory}/${:_buildout_section_name_}
mode = 0755
md5sum = 2986dcb006dc9e8508ff81f646656131
context =
section bash bash
[watcher-sigkill]
<= binsh
md5sum = 2986dcb006dc9e8508ff81f646656131
[gitlab-export]
<= binsh
md5sum = a7b32680e80f34276f0a32a5e22dad50
# macro: download a file named as section name
#
......@@ -232,7 +299,7 @@ md5sum = eb1230fee50067924ba89f4dc6e82fa9
[gitlab-parameters.cfg]
<= download-file
md5sum = 3edd435a984b51b94539ea1d4f1b3994
md5sum = a74670934ec0190cc212d1f3468c11ed
[gitlab-shell-config.yml.in]
<= download-template
......@@ -244,11 +311,15 @@ md5sum = a9cb347f60aad3465932fd36cd4fe25d
[gitlab.yml.in]
<= download-template
md5sum = 735a78d0733fd6617d3b5f3d91bfae8c
md5sum = 176939a6428a7aca4767a36421b0af2b
[instance-gitlab.cfg.in]
<= download-file
md5sum = ba83f01fd7a313b984766c50d6e48e54
md5sum = 89914e4a225f6cdebfa196d46359f6f2
[instance-gitlab-export.cfg.in]
<= download-file
md5sum = a295e524046a9e2af569ad4b55f210cd
[macrolib.cfg.in]
<= download-file
......@@ -260,11 +331,11 @@ md5sum = a11b50d2ff2b1fa842ba4aa20041e2fe
[nginx.conf.in]
<= download-template
md5sum = 7da68dba86fff79eb93c27aa1aaf1055
md5sum = 1374f38ab6f295b850d45ea0019ec05d
[rack_attack.rb.in]
<= download-template
md5sum = bc1a7c1e83b7329d97bff6724f2bec3e
md5sum = 7d0e6dc6b826f6df6b20d8574a29e2f8
[resque.yml.in]
<= download-template
......@@ -272,7 +343,7 @@ md5sum = 7c89a730889e3224548d9abe51a2d719
[smtp_settings.rb.in]
<= download-template
md5sum = d66a424516ffacea34303e2f512a7d94
md5sum = c67ea492e17f774d0e18f1217338a55f
[unicorn.rb.in]
<= download-template
......
software/gitlab/template/gitlab.yml.in
View file @ 9e01d769
......@@ -2,7 +2,7 @@
# see:
# https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example
# https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/gitlab.yml.erb
# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
{% from 'macrolib.cfg.in' import cfg, cfg_https, external_url with context %}
......@@ -86,6 +86,7 @@ production: &base
wiki: {{ cfg('default_projects_features.wiki') }}
snippets: {{ cfg('default_projects_features.snippets') }}
builds: false {# builds not supported yet <%= @gitlab_default_projects_features_builds %> #}
{# container_registry: <%= @gitlab_default_projects_features_container_registry %> #}
## Webhook settings
# Number of seconds to wait for HTTP response after sending webhook HTTP POST request (default: 10)
......@@ -145,6 +146,18 @@ production: &base
storage_path: <%= @lfs_storage_path %>
#}
{# we do not support container registry
## Container Registry
registry:
enabled: <%= @registry_enabled %>
host: <%= @registry_host %>
port: <%= @registry_port %>
api_url: <%= @registry_api_url %> # internal address to the registry, will be used by GitLab to directly communicate with API
path: <%= @registry_path %>
key: <%= @registry_key_path %>
issuer: <%= @registry_issuer %>
#}
{# we do not support Pages
## GitLab Pages (EE only)
pages:
......
software/gitlab/template/nginx.conf.in
View file @ 9e01d769
......@@ -2,7 +2,7 @@
# see:
# https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/nginx/gitlab-ssl
# https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/nginx.conf.erb
# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
{% from 'macrolib.cfg.in' import cfg with context %}
......@@ -50,9 +50,11 @@ http {
include {{ nginx_gitlab_http_conf }};
{# we don't need: ci, pages, mattermost
{# we don't need: ci, pages, mattermost, registry
include <%= @gitlab_ci_http_config %>
include <%= @gitlab_pages_http_config %>;
include <%= @gitlab_mattermost_http_config %>
include <%= @gitlab_registry_http_config %>;
#}
}
software/gitlab/template/rack_attack.rb.in
View file @ 9e01d769
......@@ -2,7 +2,7 @@
# see:
# https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/initializers/rack_attack.rb.example
# https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/rack_attack.rb.erb
# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
{% from 'macrolib.cfg.in' import cfg with context %}
......@@ -24,8 +24,9 @@ paths_to_be_protected = [
# Create one big regular expression that matches strings starting with any of
# the paths_to_be_protected.
paths_regex = Regexp.union(paths_to_be_protected.map { |path| /\A#{Regexp.escape(path)}/ })
rack_attack_enabled = Gitlab.config.rack_attack.git_basic_auth['enabled']
unless Rails.env.test?
unless Rails.env.test? || !rack_attack_enabled
Rack::Attack.throttle('protected paths', limit: {{ cfg('rate_limit_requests_per_period') }}, period: {{ cfg('rate_limit_period') }}.seconds) do |req|
if req.post? && req.path =~ paths_regex
req.ip
......
software/gitlab/template/smtp_settings.rb.in
View file @ 9e01d769
......@@ -2,7 +2,7 @@
# see:
# https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/initializers/smtp_settings.rb.sample
# https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/smtp_settings.rb.erb
# (last updated for omnibus-gitlab 8.7.9+ce.1-0-gf589ad7)
# (last updated for omnibus-gitlab 8.8.9+ce.0-g25376053)
{% from 'macrolib.cfg.in' import cfg, cfg_bool with context %}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7