Skip to content

G
gitlab-ce
Commit 567eac03 authored by Robert Speicher's avatar Robert Speicher
Browse files
Options

Reformat and copy edit the CI-to-CE migration guide

parent 0aec0d53
Hide whitespace changes
Inline Side-by-side
Showing with 231 additions and 187 deletions
doc/migrate_ci_to_ce/README.md
View file @ 567eac03
## Migrate GitLab CI to GitLab CE/EE ## Migrate GitLab CI to GitLab CE or EE
## Notice Beginning with version 8.0 of GitLab Community Edition (CE) and Enterprise
Edition (EE), GitLab CI is no longer its own application, but is instead built
into the CE and EE applications.
**You need to have working GitLab CI 7.14 to perform migration. This guide will detail the process of migrating your CI installation and data
The older versions are not supported and will most likely break migration procedure.** into your GitLab CE or EE installation.
This migration can't be done online and takes significant amount of time. ### Before we begin
Make sure to plan it ahead.
If you are running older version please follow the upgrade guide first: **You need to have a working installation of GitLab CI version 7.14 to perform
https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/7.13-to-7.14.md this migration. The older versions are not supported and will most likely break
this migration procedure.**
The migration is divided into a two parts: This migration cannot be performed online and takes a significant amount of
1. **[CI]** You will be making a changes to GitLab CI instance. time. Make sure to plan ahead.
1. **[CE]** You will be making a changes to GitLab CE/EE instance.
### 1. Stop CI server [CI] If you are running a version of GitLab CI prior to 7.14 please follow the
appropriate [update guide](https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/).
The migration is divided into three parts:
1. [GitLab CI](#part-i-gitlab-ci)
1. [Gitlab CE (or EE)](#part-ii-gitlab-ce-or-ee)
1. [Finishing Up](#part-iii-finishing-up)
### Part I: GitLab CI
#### 1. Stop GitLab CI
sudo service gitlab_ci stop sudo service gitlab_ci stop
### 2. Backup [CI]
**The migration procedure is database breaking. #### 2. Create a backup
You need to create backup if you still want to access GitLab CI in case of failure.**
The migration procedure modifies the structure of the CI database. If something
goes wrong, you will not be able to revert to a previous version without a
backup:
```bash ```bash
cd /home/gitlab_ci/gitlab-ci cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production
``` ```
### 3. Prepare GitLab CI database to migration [CI] #### 3. Rename database tables
Copy and paste the command in terminal to rename all tables. To prevent naming conflicts with database tables in GitLab CE or EE, we need to
This also breaks your database structure disallowing you to use it anymore. rename CI's tables to begin with a `ci_` prefix:
cat <<EOF | bundle exec rails dbconsole production ```sh
ALTER TABLE application_settings RENAME TO ci_application_settings; cat <<EOF | bundle exec rails dbconsole production
ALTER TABLE builds RENAME TO ci_builds; ALTER TABLE application_settings RENAME TO ci_application_settings;
ALTER TABLE commits RENAME TO ci_commits; ALTER TABLE builds RENAME TO ci_builds;
ALTER TABLE events RENAME TO ci_events; ALTER TABLE commits RENAME TO ci_commits;
ALTER TABLE jobs RENAME TO ci_jobs; ALTER TABLE events RENAME TO ci_events;
ALTER TABLE projects RENAME TO ci_projects; ALTER TABLE jobs RENAME TO ci_jobs;
ALTER TABLE runner_projects RENAME TO ci_runner_projects; ALTER TABLE projects RENAME TO ci_projects;
ALTER TABLE runners RENAME TO ci_runners; ALTER TABLE runner_projects RENAME TO ci_runner_projects;
ALTER TABLE services RENAME TO ci_services; ALTER TABLE runners RENAME TO ci_runners;
ALTER TABLE tags RENAME TO ci_tags; ALTER TABLE services RENAME TO ci_services;
ALTER TABLE taggings RENAME TO ci_taggings; ALTER TABLE tags RENAME TO ci_tags;
ALTER TABLE trigger_requests RENAME TO ci_trigger_requests; ALTER TABLE taggings RENAME TO ci_taggings;
ALTER TABLE triggers RENAME TO ci_triggers; ALTER TABLE trigger_requests RENAME TO ci_trigger_requests;
ALTER TABLE variables RENAME TO ci_variables; ALTER TABLE triggers RENAME TO ci_triggers;
ALTER TABLE web_hooks RENAME TO ci_web_hooks; ALTER TABLE variables RENAME TO ci_variables;
EOF ALTER TABLE web_hooks RENAME TO ci_web_hooks;
EOF
### 4. Remove CI cronjob ```
#### 4. Remove cronjob
``` ```
cd /home/gitlab_ci/gitlab-ci cd /home/gitlab_ci/gitlab-ci
sudo -u gitlab_ci -H bundle exec whenever --clear-crontab sudo -u gitlab_ci -H bundle exec whenever --clear-crontab
``` ```
### 5. Dump GitLab CI database [CI] #### 5. Create a database dump
First check used database and credentials on GitLab CI and GitLab CE/EE: In this step, you will need to know information about both your CI and CE (or
EE) databases, such as the server types, hosts, and ports, and the usernames and
passwords.
1. To check it on GitLab CI: We can obtain the necessary information from the `config/database.yml` files for
each installation.
cat /home/gitlab_ci/gitlab-ci/config/database.yml 1. Get the information for the CI database:
1. To check it on GitLab CE/EE:
cat /home/git/gitlab/config/database.yml
Please first check the database engine used for GitLab CI and GitLab CE/EE. ```sh
cat /home/gitlab_ci/gitlab-ci/config/database.yml
1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses it too. ```
Please follow **Dump MySQL** guide.
1. If your GitLab CI uses **postgres** and GitLab CE/EE uses **postgres**. 1. Then for the CE (or EE) database:
Please follow **Dump PostgreSQL** guide.
1. If your GitLab CI uses **mysql2** and GitLab CE/EE uses **postgres**. ```sh
Please follow **Dump MySQL and migrate to PostgreSQL** guide. cat /home/git/gitlab/config/database.yml
```
**Remember credentials stored for accessing GitLab CI. 1. The output of each command should look something like this:
You will need to put these credentials into commands executed below.**
$ cat config/database.yml [10:06:55] ```yml
#
# PRODUCTION
#
production: production:
adapter: postgresql or mysql2 adapter: postgresql (or mysql2)
encoding: utf8 encoding: utf8
reconnect: false reconnect: false
database: GITLAB_CI_DATABASE database: GITLAB_CI_DATABASE
...@@ -100,181 +109,216 @@ You will need to put these credentials into commands executed below.** ...@@ -100,181 +109,216 @@ You will need to put these credentials into commands executed below.**
host: DB_HOSTNAME host: DB_HOSTNAME
port: DB_PORT port: DB_PORT
# socket: /tmp/mysql.sock # socket: /tmp/mysql.sock
```
1. Depending on the values for `adapter`, you will have to use different
commands to perform the database dump.
**NOTE:** For any of the commands below, you'll need to substitute the
values `IN_UPPERCASE` with the corresponding values from your **CI
installation's** `config/database.yml` files above.
- If both your CI and CE (or EE) installations use **mysql2** as the `adapter`, use
`mysqldump`:
```sh
mysqldump --default-character-set=utf8 --complete-insert --no-create-info \
--host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \
ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql
```
#### a. Dump MySQL - If both your CI and CE (or EE) installations use **postgresql** as the
`adapter`, use `pg_dump`:
mysqldump --default-character-set=utf8 --complete-insert --no-create-info \
--host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p ```sh
GITLAB_CI_DATABASE \ pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT \
ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql
ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ ```
ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql
- If your CI installation uses **mysql2** as the `adapter` and your CE (or
#### b. Dump PostgreSQL EE) installation uses **postgresql**, use `mysqldump` to dump the database
and then convert it to PostgreSQL using [mysql-postgresql-converter]:
pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql
```sh
#### c. Dump MySQL and migrate to PostgreSQL # Dump existing MySQL database first
mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \
# Dump existing MySQL database first --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p GITLAB_CI_DATABASE \
mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
--host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \
GITLAB_CI_DATABASE \ ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp
ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \
ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ # Convert database to be compatible with PostgreSQL
ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab
python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2
# Convert database to be compatible with PostgreSQL ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed
git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab
python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2 # Filter to only include INSERT statements
ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql
```
# Filter to only include INSERT statements
grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql [mysql-postgresql-converter]: https://github.com/gitlabhq/mysql-postgresql-converter
### 6. Make sure that your GitLab CE/EE is 8.0 [CE] ### Part II: GitLab CE (or EE)
Please verify that you use GitLab CE/EE 8.0. #### 1. Ensure GitLab is updated
If not, please follow the update guide: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md
Your GitLab CE or EE installation **must be version 8.0**. If it's not, follow
### 7. Stop GitLab CE/EE [CE] the [update guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md).
Before you can migrate data you need to stop GitLab CE/EE first. #### 2. Stop GitLab
Before you can migrate data you need to stop the GitLab service first:
sudo service gitlab stop sudo service gitlab stop
### 8. Backup GitLab CE/EE [CE]
This migration poses a **significant risk** of breaking your GitLab CE/EE. #### 3. Create a backup
**You should create the GitLab CI/EE backup before doing it.**
This migration poses a **significant risk** of breaking your GitLab
installation. Create a backup before proceeding:
cd /home/git/gitlab cd /home/git/gitlab
sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production
### 9. Copy secret tokens [CE] #### 4. Copy secret tokens from CI
The `secrets.yml` file stores encryption keys for secure variables. The `secrets.yml` file stores encryption keys for secure variables.
You need to copy the content of `config/secrets.yml` to the same file in GitLab CE. You need to copy the contents of GitLab CI's `config/secrets.yml` file to the
same file in GitLab CE:
sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml
sudo chown git:git /home/git/gitlab/config/secrets.yml sudo chown git:git /home/git/gitlab/config/secrets.yml
sudo chown 0600 /home/git/gitlab/config/secrets.yml sudo chown 0600 /home/git/gitlab/config/secrets.yml
### 10. New configuration options for `gitlab.yml` [CE]
There are new configuration options available for [`gitlab.yml`](config/gitlab.yml.example). #### 5. New configuration options for `gitlab.yml`
View them with the command below and apply them manually to your current `gitlab.yml`:
There are new configuration options available for `gitlab.yml`. View them with
the command below and apply them manually to your current `gitlab.yml`:
```sh ```sh
git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example git diff origin/7-14-stable:config/gitlab.yml.example origin/8-0-stable:config/gitlab.yml.example
``` ```
The new options include configuration of GitLab CI that are now being part of GitLab CE and EE. The new options include configuration settings for GitLab CI.
### 11. Copy build logs [CE] #### 6. Copy build logs
You need to copy the contents of `builds/` to the same directory in GitLab CE/EE. You need to copy the contents of GitLab CI's `builds/` directory to the
corresponding directory in GitLab CE or EE:
sudo rsync -av /home/gitlab_ci/gitlab-ci/builds /home/git/gitlab/builds sudo rsync -av /home/gitlab_ci/gitlab-ci/builds /home/git/gitlab/builds
sudo chown -R git:git /home/git/gitlab/builds sudo chown -R git:git /home/git/gitlab/builds
The build traces are usually quite big so it will take a significant amount of time. The build logs are usually quite big so it may take a significant amount of
time.
### 12. Import GitLab CI database [CE] #### 7. Import GitLab CI database
The one of the last steps is to import existing GitLab CI database. Now you'll import the GitLab CI database dump that you [created
earlier](#create-a-database-dump) into the GitLab CE or EE database:
sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql
sudo chown git:git /home/git/gitlab/gitlab_ci.sql sudo chown git:git /home/git/gitlab/gitlab_ci.sql
sudo -u git -H bundle exec rake ci:migrate CI_DUMP=/home/git/gitlab/gitlab_ci.sql RAILS_ENV=production sudo -u git -H bundle exec rake ci:migrate CI_DUMP=/home/git/gitlab/gitlab_ci.sql RAILS_ENV=production
The task does: This task will:
1. Delete data from all existing CI tables 1. Delete data from all existing CI tables
1. Import database data 1. Import data from database dump
1. Fix database auto increments 1. Fix database auto-increments
1. Fix tags assigned to Builds and Runners 1. Fix tags assigned to Builds and Runners
1. Fix services used by CI 1. Fix services used by CI
### 13. Start GitLab [CE] #### 8. Start GitLab
You can start GitLab CI/EE now and see if everything is working. You can start GitLab CI (or EE) now and see if everything is working:
sudo service gitlab start sudo service gitlab start
### 14. Update nginx [CI] ### Part III: Finishing Up
Now get back to GitLab CI and update **Nginx** configuration in order to: #### 1. Update Nginx configuration
1. Have all existing runners able to communicate with a migrated GitLab CI.
1. Have GitLab able send build triggers to CI address specified in Project's settings -> Services -> GitLab CI. To ensure that your existing CI runners are able to communicate with the
migrated installation, and that existing build triggers still work, you'll need
You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste: to update your Nginx configuration to redirect requests for the old locations to
the new ones.
# GITLAB CI
server { Edit `/etc/nginx/sites-available/gitlab_ci` and paste:
listen 80 default_server; # e.g., listen 192.168.1.1:80;
server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com; ```nginx
# GITLAB CI
access_log /var/log/nginx/gitlab_ci_access.log; server {
error_log /var/log/nginx/gitlab_ci_error.log; listen 80 default_server; # e.g., listen 192.168.1.1:80;
server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com;
# expose API to fix runners
location /api { access_log /var/log/nginx/gitlab_ci_access.log;
proxy_read_timeout 300; error_log /var/log/nginx/gitlab_ci_error.log;
proxy_connect_timeout 300;
proxy_redirect off; # expose API to fix runners
proxy_set_header X-Real-IP $remote_addr; location /api {
proxy_read_timeout 300;
# You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN proxy_connect_timeout 300;
resolver 8.8.8.8 8.8.4.4; proxy_redirect off;
proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; proxy_set_header X-Real-IP $remote_addr;
}
# You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
# expose build endpoint to allow trigger builds resolver 8.8.8.8 8.8.4.4;
location ~ ^/projects/\d+/build$ { proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
proxy_read_timeout 300; }
proxy_connect_timeout 300;
proxy_redirect off; # expose build endpoint to allow trigger builds
proxy_set_header X-Real-IP $remote_addr; location ~ ^/projects/\d+/build$ {
proxy_read_timeout 300;
# You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN proxy_connect_timeout 300;
resolver 8.8.8.8 8.8.4.4; proxy_redirect off;
proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; proxy_set_header X-Real-IP $remote_addr;
}
# You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN
# redirect all other CI requests resolver 8.8.8.8 8.8.4.4;
location / { proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; }
}
# redirect all other CI requests
# adjust this to match the largest build log your runners might submit, location / {
# set to 0 to disable limit return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri;
client_max_body_size 10m; }
}
# adjust this to match the largest build log your runners might submit,
Make sure to fill the blanks to match your setup: # set to 0 to disable limit
1. **YOUR_CI_SERVER_FQDN**: The existing public facing address of GitLab CI, eg. ci.gitlab.com. client_max_body_size 10m;
1. **YOUR_GITLAB_SERVER_FQDN**: The public facing address of GitLab CE/EE, eg. gitlab.com. }
```
**Make sure to not remove the `/ci$request_uri`. This is required to properly forward the requests.**
Make sure you substitute these placeholder values with your real ones:
You should also make sure that you can do:
1. `YOUR_CI_SERVER_FQDN`: The existing public-facing address of your GitLab CI
install (e.g., `ci.gitlab.com`).
1. `YOUR_GITLAB_SERVER_FQDN`: The current public-facing address of your GitLab
CE (or EE) install (e.g., `gitlab.com`).
**Make sure not to remove the `/ci$request_uri` part. This is required to properly forward the requests.**
You should also make sure that you can:
1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server. 1. `curl https://YOUR_GITLAB_SERVER_FQDN/` from your previous GitLab CI server.
1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE/EE server. 1. `curl https://YOUR_CI_SERVER_FQDN/` from your GitLab CE (or EE) server.
## Check your configuration #### 2. Check Nginx configuration
sudo nginx -t sudo nginx -t
## Restart nginx #### 3. Restart Nginx
sudo /etc/init.d/nginx restart sudo /etc/init.d/nginx restart
### 15. Done! #### 4. Done!
If everything went OK you should be able to access all your GitLab CI data by pointing your browser to: If everything went well you should be able to access your migrated CI install by
https://gitlab.example.com/ci/. visiting `https://gitlab.example.com/ci/`.
The GitLab CI should also work when using the previous address, redirecting you to the GitLab CE/EE. If you visit the old GitLab CI address, you should be redirected to the new one.
**Enjoy!** **Enjoy!**
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7