Skip to content

G
gitlab-ce
Commit eb5fd3f8 authored by Robert Speicher's avatar Robert Speicher
Browse files
Options

Merge branch 'rs-ci-migration-guide' into 'master' 

Reformat and copy edit the CI-to-CE migration guide

[ci skip]

See merge request !1335
parents 34e43c1a 524381dc
Show whitespace changes
Inline Side-by-side
Showing with 264 additions and 220 deletions
doc/migrate_ci_to_ce/README.md
View file @ eb5fd3f8
## Migrate GitLab CI to GitLab CE/EE ## Migrate GitLab CI to GitLab CE or EE
## Notice Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise
Edition (EE), GitLab CI is no longer its own application, but is instead built
into the CE and EE applications.
**You need to have working GitLab CI 7.14 to perform migration. This guide will detail the process of migrating your CI installation and data
The older versions are not supported and will most likely break migration procedure.** into your GitLab CE or EE installation.
This migration can't be done online and takes significant amount of time. ### Before we begin
Make sure to plan it ahead.
If you are running older version please follow the upgrade guide first: **You need to have a working installation of GitLab CI version 8.0 to perform
https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/7.13-to-7.14.md this migration. The older versions are not supported and will most likely break
this migration procedure.**
The migration is divided into a two parts: This migration cannot be performed online and takes a significant amount of
1. **[CI]** You will be making a changes to GitLab CI instance. time. Make sure to plan ahead.
1. **[CE]** You will be making a changes to GitLab CE/EE instance.
### 1. Stop CI server [CI] If you are running a version of GitLab CI prior to 8.0 please follow the
appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/tree/master/doc/update/)
before proceeding.
sudo service gitlab_ci stop The migration is divided into four parts and covers both manual and Omnibus
installations:
### 2. Backup [CI] 1. [GitLab CI](#part-i-gitlab-ci)
1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee)
1. [Nginx configuration](#part-iii-nginx-configuration)
1. [Finishing Up](#part-iv-finishing-up)
**The migration procedure is database breaking. ### Part I: GitLab CI
You need to create backup if you still want to access GitLab CI in case of failure.**
```bash #### 1. Stop GitLab CI
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production
```
### 3. Prepare GitLab CI database to migration [CI] # Manual installation
sudo service gitlab_ci stop
Copy and paste the command in terminal to rename all tables.
This also breaks your database structure disallowing you to use it anymore.
cat <<EOF | bundle exec rails dbconsole production
ALTER TABLE application_settings RENAME TO ci_application_settings;
ALTER TABLE builds RENAME TO ci_builds;
ALTER TABLE commits RENAME TO ci_commits;
ALTER TABLE events RENAME TO ci_events;
ALTER TABLE jobs RENAME TO ci_jobs;
ALTER TABLE projects RENAME TO ci_projects;
ALTER TABLE runner_projects RENAME TO ci_runner_projects;
ALTER TABLE runners RENAME TO ci_runners;
ALTER TABLE services RENAME TO ci_services;
ALTER TABLE tags RENAME TO ci_tags;
ALTER TABLE taggings RENAME TO ci_taggings;
ALTER TABLE trigger_requests RENAME TO ci_trigger_requests;
ALTER TABLE triggers RENAME TO ci_triggers;
ALTER TABLE variables RENAME TO ci_variables;
ALTER TABLE web_hooks RENAME TO ci_web_hooks;
EOF
### 4. Remove CI cronjob
``` # Omnibus installation
cd /home/gitlab_ci/gitlab-ci sudo gitlab-ctl stop ci-unicorn ci-sidekiq
sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
```
### 5. Dump GitLab CI database [CI] #### 2. Create a backup
First check used database and credentials on GitLab CI and GitLab CE/EE: The migration procedure modifies the structure of the CI database. If something
goes wrong, you will not be able to revert to a previous version without a
backup.
1. To check it on GitLab CI: If your GitLab CI installation uses **MySQL** and your GitLab CE (or EE)
installation uses **PostgreSQL** you'll need to convert the CI database by
setting a `MYSQL_TO_POSTGRESQL` flag.
cat /home/gitlab_ci/gitlab-ci/config/database.yml If you use the Omnibus package you most likely use **PostgreSQL** on both GitLab
CE (or EE) and CI.
1. To check it on GitLab CE/EE: You can check which database each install is using by viewing their
database configuration files:
cat /home/gitlab_ci/gitlab-ci/config/database.yml
cat /home/git/gitlab/config/database.yml cat /home/git/gitlab/config/database.yml
Please first check the database engine used for GitLab CI and GitLab CE/EE. - If both applications use the same database `adapter`, create the backup with
this command:
1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses it too. # Manual installation
Please follow **Dump MySQL** guide. cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production
1. If your GitLab CI uses **postgres** and GitLab CE/EE uses **postgres**. # Omnibus installation
Please follow **Dump PostgreSQL** guide. sudo gitlab-ci-rake backup:create
1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses **postgres**. - If CI uses MySQL, and CE (or EE) uses PostgreSQL, create the backup with this
Please follow **Dump MySQL and migrate to PostgreSQL** guide. command (note the `MYSQL_TO_POSTGRESQL` flag):
**Remember credentials stored for accessing GitLab CI. # Manual installation
You will need to put these credentials into commands executed below.** cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production MYSQL_TO_POSTGRESQL=1
$ cat config/database.yml [10:06:55] # Omnibus installation
# sudo gitlab-ci-rake backup:create MYSQL_TO_POSTGRESQL=1
# PRODUCTION
#
production:
adapter: postgresql or mysql2
encoding: utf8
reconnect: false
database: GITLAB_CI_DATABASE
pool: 5
username: DB_USERNAME
password: DB_PASSWORD
host: DB_HOSTNAME
port: DB_PORT
# socket: /tmp/mysql.sock
#### a. Dump MySQL #### 3. Remove cronjob
mysqldump --default-character-set=utf8 --complete-insert --no-create-info \ **Note:** This step is only required for **manual installations**. Omnibus users
--host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p can [skip to the next step](#part-ii-gitlab-ce-or-ee).
GITLAB_CI_DATABASE \
ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql
#### b. Dump PostgreSQL # Manual installation
cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql ### Part II: GitLab CE (or EE)
#### c. Dump MySQL and migrate to PostgreSQL #### 1. Ensure GitLab is updated
# Dump existing MySQL database first Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow
mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md)
--host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p before proceeding.
GITLAB_CI_DATABASE \
ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp
# Convert database to be compatible with PostgreSQL If you use the Omnibus packages simply run `apt-get upgrade` to install the
git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab latest version.
python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2
ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed
# Filter to only include INSERT statements #### 2. Prevent CI usage during the migration process
grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql
### 6. Make sure that your GitLab CE/EE is 8.0 [CE] As an administrator, go to **Admin Area** -> **Settings**, and under **Continuous
Integration** uncheck **Disable to prevent CI usage until rake ci:migrate is run
(8.0 only)**.
Please verify that you use GitLab CE/EE 8.0. This will disable the CI integration and prevent users from creating CI projects
If not, please follow the update guide: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md until the migration process is completed.
### 7. Stop GitLab CE/EE [CE] #### 3. Stop GitLab
Before you can migrate data you need to stop GitLab CE/EE first. Before you can migrate data you need to stop the GitLab service first:
# Manual installation
sudo service gitlab stop sudo service gitlab stop
### 8. Backup GitLab CE/EE [CE] # Omnibus installation
sudo gitlab-ctl stop unicorn sidekiq
#### 4. Create a backup
This migration poses a **significant risk** of breaking your GitLab CE/EE. This migration poses a **significant risk** of breaking your GitLab
**You should create the GitLab CI/EE backup before doing it.** installation. Create a backup before proceeding:
# Manual installation
cd /home/git/gitlab cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
### 9. Copy secret tokens [CE] # Omnibus installation
sudo gitlab-rake gitlab:backup:create
It's possible to speed up backup creation by skipping repositories and uploads:
# Manual installation
cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production SKIP=repositories,uploads
# Omnibus installation
sudo gitlab-rake gitlab:backup:create SKIP=repositories,uploads
#### 5. Copy secret tokens from CI
The `secrets.yml` file stores encryption keys for secure variables. The `secrets.yml` file stores encryption keys for secure variables.
You need to copy the content of `config/secrets.yml` to the same file in GitLab CE. - **Manual installations** need to copy the contents of GitLab CI's
`config/secrets.yml` file to the same file in GitLab CE:
```bash
# Manual installation
sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml
sudo chown git:git /home/git/gitlab/config/secrets.yml sudo chown git:git /home/git/gitlab/config/secrets.yml
sudo chown 0600 /home/git/gitlab/config/secrets.yml sudo chown 0600 /home/git/gitlab/config/secrets.yml
```
- **Omnibus installations** where GitLab CI and CE (or EE) are on the same
server don't need to do anything further, because the secrets are stored in
`/etc/gitlab/gitlab-secrets.json`.
- **Omnibus installations** where GitLab CI is on a different server than CE (or
EE) will need to:
1. On the CI server, copy the `db_key_base` value from
`/etc/gitlab/gitlab-secrets.json`
1. On the CE (or EE) server, add `gitlab_ci['db_key_base'] =
"VALUE_FROM_ABOVE"` to the `/etc/gitlab/gitlab.rb` file and run `sudo
gitlab-ctl reconfigure`
#### 6. New configuration options for `gitlab.yml`
**Note:** This step is only required for **manual installations**. Omnibus users
can [skip to the next step](#7-copy-backup-from-gitlab-ci).
### 10. New configuration options for `gitlab.yml` [CE] There are new configuration options available for `gitlab.yml`. View them with
the command below and apply them manually to your current `gitlab.yml`:
There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example
View them with the command below and apply them manually to your current `gitlab.yml`:
```sh The new options include configuration settings for GitLab CI.
git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example
#### 7. Copy backup from GitLab CI
```bash
# Manual installation
sudo cp -v /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar /home/git/gitlab/tmp/backups
sudo chown git:git /home/git/gitlab/tmp/backups/*_gitlab_ci_backup.tar
# Omnibus installation
sudo cp -v /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar /var/opt/gitlab/backups/
sudo chown git:git /var/opt/gitlab/backups/*_gitlab_ci_backup.tar
``` ```
The new options include configuration of GitLab CI that are now being part of GitLab CE and EE. If moving across the servers you can use `scp`.
However, this requires you to provide an authorized key or password to login to
the GitLab CE (or EE) server from the CI server. You can try to use ssh-agent
from your local machine to have that: login to your GitLab CI server using
`ssh -A`.
### 11. Copy build logs [CE] ```bash
# Manual installation
scp /home/gitlab_ci/gitlab-ci/tmp/backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/home/git/gitlab/tmp/backup
You need to copy the contents of `builds/` to the same directory in GitLab CE/EE. # Omnibus installation
scp /var/opt/gitlab/ci-backups/*_gitlab_ci_backup.tar root@gitlab.example.com:/var/opt/gitlab/backups/
```
sudo rsync -av /home/gitlab_ci/gitlab-ci/builds /home/git/gitlab/builds #### 8. Import GitLab CI backup
sudo chown -R git:git /home/git/gitlab/builds
The build traces are usually quite big so it will take a significant amount of time. Now you'll import the GitLab CI database dump that you created earlier into the
GitLab CE or EE database:
### 12. Import GitLab CI database [CE] # Manual installation
sudo -u git -H bundle exec rake ci:migrate RAILS_ENV=production
The one of the last steps is to import existing GitLab CI database. # Omnibus installation
sudo gitlab-rake ci:migrate
sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql This task will take some time.
sudo chown git:git /home/git/gitlab/gitlab_ci.sql
sudo -u git -H bundle exec rake ci:migrate CI_DUMP=/home/git/gitlab/gitlab_ci.sql RAILS_ENV=production
The task does: This migration task automatically re-enables the CI setting that you
1. Delete data from all existing CI tables [disabled earlier](#2-prevent-ci-usage-during-the-migration-process).
1. Import database data
1. Fix database auto increments
1. Fix tags assigned to Builds and Runners
1. Fix services used by CI
### 13. Start GitLab [CE] #### 9. Start GitLab
You can start GitLab CI/EE now and see if everything is working. You can start GitLab CE (or EE) now and see if everything is working:
# Manual installation
sudo service gitlab start sudo service gitlab start
### 14. Update nginx [CI] # Omnibus installation
sudo gitlab-ctl restart unicorn sidekiq
### Part III: Nginx configuration
This section is only required for **manual installations**. Omnibus users can
[skip to the final step](#part-iv-finishing-up).
#### 1. Update Nginx configuration
Now get back to GitLab CI and update **Nginx** configuration in order to: To ensure that your existing CI runners are able to communicate with the
1. Have all existing runners able to communicate with a migrated GitLab CI. migrated installation, and that existing build triggers still work, you'll need
1. Have GitLab able send build triggers to CI address specified in Project's settings -> Services -> GitLab CI. to update your Nginx configuration to redirect requests for the old locations to
the new ones.
You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste: Edit `/etc/nginx/sites-available/gitlab_ci` and paste:
# GITLAB CI ```nginx
server { # GITLAB CI
server {
listen 80 default_server; # e.g., listen 192.168.1.1:80; listen 80 default_server; # e.g., listen 192.168.1.1:80;
server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com; server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com;
...@@ -250,31 +282,43 @@ You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste: ...@@ -250,31 +282,43 @@ You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste:
# adjust this to match the largest build log your runners might submit, # adjust this to match the largest build log your runners might submit,
# set to 0 to disable limit # set to 0 to disable limit
client_max_body_size 10m; client_max_body_size 10m;
} }
```
Make sure you substitute these placeholder values with your real ones:
1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI
install (e.g., `ci.gitlab.com`).
1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab
CE (or EE) install (e.g., `gitlab.com`).
Make sure to fill the blanks to match your setup: **Make sure not to remove the `/ci$request_uri` part. This is required to
1. **YOUR_CI_SERVER_FQDN**: The existing public facing address of GitLab CI, eg. ci.gitlab.com. properly forward the requests.**
1. **YOUR_GITLAB_SERVER_FQDN**: The public facing address of GitLab CE/EE, eg. gitlab.com.
**Make sure to not remove the `/ci$request_uri`. This is required to properly forward the requests.** You should also make sure that you can:
You should also make sure that you can do:
1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server. 1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server.
1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE/EE server. 1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE (or EE) server.
## Check your configuration #### 2. Check Nginx configuration
sudo nginx -t sudo nginx -t
## Restart nginx #### 3. Restart Nginx
sudo /etc/init.d/nginx restart sudo /etc/init.d/nginx restart
### 15. Done! ### Part IV: Finishing Up
If everything went OK you should be able to access all your GitLab CI data by pointing your browser to: If everything went well you should be able to access your migrated CI install by
https://gitlab.example.com/ci/. visiting `https://gitlab.example.com/ci/`. If you visit the old GitLab CI
address, you should be redirected to the new one.
The GitLab CI should also work when using the previous address, redirecting you to the GitLab CE/EE.
**Enjoy!** **Enjoy!**
### Troubleshooting
#### Restore from backup
If something went wrong and you need to restore a backup, consult the [Backup
restoration](../raketasks/backup_restore.md) guide.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7