Commit 7fcd386a authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Clean up the mysql to postgres guide

This change refactors the pgloader instructions to have
less headings and be more straightforward.
parent 14bbc936
--- ---
last_updated: 2018-02-07 last_updated: 2019-03-27
--- ---
# Migrating from MySQL to PostgreSQL # Migrating from MySQL to PostgreSQL
> **Note:** This guide assumes you have a working GitLab instance with This guide documents how to take a working GitLab instance that uses MySQL and
> MySQL and want to migrate to bundled PostgreSQL database. migrate it to a PostgreSQL database.
## Omnibus installation ## Requirements
### Prerequisites [pgloader](http://pgloader.io) 3.4.1+ is required.
First, we'll need to enable the bundled PostgreSQL database with up-to-date You can install it directly from your distribution, for example in
schema. Next, we'll use [pgloader](http://pgloader.io) to migrate the data Debian/Ubuntu:
from the old MySQL database to the new PostgreSQL one.
Here's what you'll need to have installed: 1. Search for the version:
- pgloader 3.4.1+ ```bash
- Omnibus GitLab apt-cache madison pgloader
- MySQL ```
1. If the version is 3.4.1+, install it with:
```bash
sudo apt-get install pgloader
```
### Enable bundled PostgreSQL database If your distribution's version is too old, use PostgreSQL's repository:
```bash
# Add repository
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
# Add key
sudo apt-get install wget ca-certificates
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# Install package
sudo apt-get update
sudo apt-get install pgloader
```
For other distributions, follow the instructions in PostrgreSQL's
[download page](https://www.postgresql.org/download/) to add their repository
and then install `pgloader`.
## Omnibus GitLab installations
For [Omnibus GitLab packages](https://about.gitlab.com/install/), you'll first
need to enable the bundled PostgreSQL:
1. Stop GitLab: 1. Stop GitLab:
``` bash ```bash
sudo gitlab-ctl stop sudo gitlab-ctl stop
``` ```
...@@ -40,39 +67,34 @@ Here's what you'll need to have installed: ...@@ -40,39 +67,34 @@ Here's what you'll need to have installed:
and alike. You could just comment all of them out so that we'll just use and alike. You could just comment all of them out so that we'll just use
the defaults. the defaults.
1. [Reconfigure GitLab] for the changes to take effect: 1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure)
for the changes to take effect.
``` bash
sudo gitlab-ctl reconfigure
```
1. Start Unicorn and PostgreSQL so that we can prepare the schema: 1. Start Unicorn and PostgreSQL so that we can prepare the schema:
``` bash ```bash
sudo gitlab-ctl start unicorn sudo gitlab-ctl start unicorn
sudo gitlab-ctl start postgresql sudo gitlab-ctl start postgresql
``` ```
1. Run the following commands to prepare the schema: 1. Run the following commands to prepare the schema:
``` bash ```bash
sudo gitlab-rake db:create db:migrate sudo gitlab-rake db:create db:migrate
``` ```
1. Stop Unicorn to prevent other database access from interfering with the loading of data: 1. Stop Unicorn to prevent other database access from interfering with the loading of data:
``` bash ```bash
sudo gitlab-ctl stop unicorn sudo gitlab-ctl stop unicorn
``` ```
After these steps, you'll have a fresh PostgreSQL database with up-to-date schema. After these steps, you'll have a fresh PostgreSQL database with up-to-date schema.
### Migrate data from MySQL to PostgreSQL Next, we'll use `pgloader` to migrate the data from the old MySQL database to the
new PostgreSQL one:
Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
1. Save the following snippet in a `commands.load` file, and edit with your 1. Save the following snippet in a `commands.load` file, and edit with your
database `username`, `password` and `host`: MySQL database `username`, `password` and `host`:
``` ```
LOAD DATABASE LOAD DATABASE
...@@ -90,7 +112,7 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: ...@@ -90,7 +112,7 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
1. Start the migration: 1. Start the migration:
``` bash ```bash
sudo -u gitlab-psql pgloader commands.load sudo -u gitlab-psql pgloader commands.load
``` ```
...@@ -117,106 +139,56 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: ...@@ -117,106 +139,56 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
Total import time 1894 1894 0 12.497s Total import time 1894 1894 0 12.497s
``` ```
If there is no output for more than 30 minutes, it's possible pgloader encountered an error. See If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See
the [troubleshooting guide](#Troubleshooting) for more details. the [troubleshooting guide](#troubleshooting) for more details.
1. Start GitLab: 1. Start GitLab:
``` bash ```bash
sudo gitlab-ctl start sudo gitlab-ctl start
``` ```
Now, you can verify that everything worked by visiting GitLab. You can now verify that everything works as expected by visiting GitLab.
### Troubleshooting ## Source installations
#### Permissions For installations from source that use MySQL, you'll first need to
[install PostgreSQL and create a database](../install/installation.md#6-database).
Note that the PostgreSQL user that you use for the above MUST have **superuser** privileges. Otherwise, you may see After the database is created, go on with the following steps:
a similar message to the following:
```
debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread
#<THREAD "lparallel" RUNNING {10078A3513}>:
Database error 42501: permission denied: "RI_ConstraintTrigger_a_20937" is a system trigger
QUERY: ALTER TABLE ci_builds DISABLE TRIGGER ALL;
2017-08-23T00:36:56.782000Z ERROR Database error 42501: permission denied: "RI_ConstraintTrigger_c_20864" is a system trigger
QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL;
```
#### Experiencing 500 errors after the migration
If you experience 500 errors after the migration, try to clear the cache:
``` bash
sudo gitlab-rake cache:clear
```
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
## Source installation
### Prerequisites
#### Install PostgreSQL and create database
See [installation guide](../install/installation.md#6-database).
#### Install [pgloader](http://pgloader.io) 3.4.1+
Install directly from your distro:
``` bash
sudo apt-get install pgloader
```
If this version is too old, use PostgreSQL's repository:
``` bash
# add repository
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
# add key
sudo apt-get install wget ca-certificates
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
# install package
sudo apt-get update
sudo apt-get install pgloader
```
### Enable bundled PostgreSQL database
1. Stop GitLab: 1. Stop GitLab:
``` bash ```bash
sudo service gitlab stop sudo service gitlab stop
``` ```
1. Switch database from MySQL to PostgreSQL 1. Switch database from MySQL to PostgreSQL
``` bash ```bash
cd /home/git/gitlab cd /home/git/gitlab
sudo -u git mv config/database.yml config/database.yml.bak sudo -u git mv config/database.yml config/database.yml.bak
sudo -u git cp config/database.yml.postgresql config/database.yml sudo -u git cp config/database.yml.postgresql config/database.yml
sudo -u git -H chmod o-rwx config/database.yml sudo -u git -H chmod o-rwx config/database.yml
``` ```
1. Install Gems related to Postgresql 1. Install Gems related to Postgresql
``` bash ```bash
sudo -u git -H rm .bundle/config sudo -u git -H rm .bundle/config
sudo -u git -H bundle install --deployment --without development test mysql aws kerberos sudo -u git -H bundle install --deployment --without development test mysql aws kerberos
``` ```
1. Run the following commands to prepare the schema: 1. Run the following commands to prepare the schema:
``` bash ```bash
sudo -u git -H bundle exec rake db:create db:migrate RAILS_ENV=production sudo -u git -H bundle exec rake db:create db:migrate RAILS_ENV=production
``` ```
After these steps, you'll have a fresh PostgreSQL database with up-to-date schema. After these steps, you'll have a fresh PostgreSQL database with up-to-date schema.
### Migrate data from MySQL to PostgreSQL Next, we'll use `pgloader` to migrate the data from the old MySQL database to the
new PostgreSQL one:
Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
1. Save the following snippet in a `commands.load` file, and edit with your 1. Save the following snippet in a `commands.load` file, and edit with your
MySQL `username`, `password` and `host`: MySQL `username`, `password` and `host`:
...@@ -237,7 +209,7 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: ...@@ -237,7 +209,7 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
1. Start the migration: 1. Start the migration:
``` bash ```bash
sudo -u postgres pgloader commands.load sudo -u postgres pgloader commands.load
``` ```
...@@ -264,23 +236,43 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: ...@@ -264,23 +236,43 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL:
Total import time 1894 1894 0 12.497s Total import time 1894 1894 0 12.497s
``` ```
If there is no output for more than 30 minutes, it's possible pgloader encountered an error. See If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See
the [troubleshooting guide](#Troubleshooting) for more details. the [troubleshooting guide](#troubleshooting) for more details.
1. Start GitLab: 1. Start GitLab:
``` bash ```bash
sudo service gitlab start sudo service gitlab start
``` ```
Now, you can verify that everything worked by visiting GitLab. You can now verify that everything works as expected by visiting GitLab.
### Troubleshooting ## Troubleshooting
#### Experiencing 500 errors after the migration Sometimes, you might encounter some errors during or after the migration.
### Database error permission denied
The PostgreSQL user that you use for the migration MUST have **superuser** privileges.
Otherwise, you may see a similar message to the following:
```
debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread
#<THREAD "lparallel" RUNNING {10078A3513}>:
Database error 42501: permission denied: "RI_ConstraintTrigger_a_20937" is a system trigger
QUERY: ALTER TABLE ci_builds DISABLE TRIGGER ALL;
2017-08-23T00:36:56.782000Z ERROR Database error 42501: permission denied: "RI_ConstraintTrigger_c_20864" is a system trigger
QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL;
```
### Experiencing 500 errors after the migration
If you experience 500 errors after the migration, try to clear the cache: If you experience 500 errors after the migration, try to clear the cache:
``` bash ```bash
# Omnibus GitLab
sudo gitlab-rake cache:clear
# Installations from source
sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production
``` ```
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment