Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
241f83f8
Commit
241f83f8
authored
Aug 11, 2020
by
Ben Bodenmiller
Committed by
Achilleas Pipinellis
Aug 11, 2020
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Moving repositories formatting improvements
parent
f3ea409a
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
24 additions
and
23 deletions
+24
-23
doc/administration/operations/moving_repositories.md
doc/administration/operations/moving_repositories.md
+24
-23
No files found.
doc/administration/operations/moving_repositories.md
View file @
241f83f8
...
@@ -9,16 +9,17 @@ We will look at three scenarios: the target directory is empty, the
...
@@ -9,16 +9,17 @@ We will look at three scenarios: the target directory is empty, the
target directory contains an outdated copy of the repositories, and
target directory contains an outdated copy of the repositories, and
how to deal with thousands of repositories.
how to deal with thousands of repositories.
**
Each of the approaches we list can/will overwrite data in the
DANGER:
**Danger:**
Each of the approaches we list can/will overwrite data in the
target directory
`/mnt/gitlab/repositories`
. Do not mix up the
target directory
`/mnt/gitlab/repositories`
. Do not mix up the
source and the target.
**
source and the target.
## Target directory is empty: use a
tar
pipe
## Target directory is empty: use a
`tar`
pipe
If the target directory
`/mnt/gitlab/repositories`
is empty the
If the target directory
`/mnt/gitlab/repositories`
is empty the
simplest thing to do is to use a
tar
pipe. This method has low
simplest thing to do is to use a
`tar`
pipe. This method has low
overhead and
tar
is almost always already installed on your system.
overhead and
`tar`
is almost always already installed on your system.
However, it is not possible to resume an interrupted
tar
pipe: if
However, it is not possible to resume an interrupted
`tar`
pipe: if
that happens then all data must be copied again.
that happens then all data must be copied again.
```
shell
```
shell
...
@@ -28,9 +29,9 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
...
@@ -28,9 +29,9 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
If you want to see progress, replace
`-xf`
with
`-xvf`
.
If you want to see progress, replace
`-xf`
with
`-xvf`
.
###
Tar
pipe to another server
###
`tar`
pipe to another server
You can also use a
tar
pipe to copy data to another server. If your
You can also use a
`tar`
pipe to copy data to another server. If your
`git`
user has SSH access to the new server as
`git@newserver`
, you
`git`
user has SSH access to the new server as
`git@newserver`
, you
can pipe the data through SSH.
can pipe the data through SSH.
...
@@ -42,13 +43,13 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
...
@@ -42,13 +43,13 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\
If you want to compress the data before it goes over the network
If you want to compress the data before it goes over the network
(which will cost you CPU cycles) you can replace
`ssh`
with
`ssh -C`
.
(which will cost you CPU cycles) you can replace
`ssh`
with
`ssh -C`
.
## The target directory contains an outdated copy of the repositories: use
rsync
## The target directory contains an outdated copy of the repositories: use
`rsync`
If the target directory already contains a partial / outdated copy
If the target directory already contains a partial / outdated copy
of the repositories it may be wasteful to copy all the data again
of the repositories it may be wasteful to copy all the data again
with
tar. In this scenario it is better to use rsync
. This utility
with
`tar`
. In this scenario it is better to use
`rsync`
. This utility
is either already installed on your system or easily installable
is either already installed on your system or easily installable
via
apt, yum
etc.
via
`apt`
,
`yum`
,
etc.
```
shell
```
shell
sudo
-u
git sh
-c
'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
sudo
-u
git sh
-c
'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
...
@@ -59,30 +60,30 @@ The `/.` in the command above is very important, without it you can
...
@@ -59,30 +60,30 @@ The `/.` in the command above is very important, without it you can
easily get the wrong directory structure in the target directory.
easily get the wrong directory structure in the target directory.
If you want to see progress, replace
`-a`
with
`-av`
.
If you want to see progress, replace
`-a`
with
`-av`
.
### Single
rsync
to another server
### Single
`rsync`
to another server
If the
`git`
user on your source system has SSH access to the target
If the
`git`
user on your source system has SSH access to the target
server you can send the repositories over the network with
rsync
.
server you can send the repositories over the network with
`rsync`
.
```
shell
```
shell
sudo
-u
git sh
-c
'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
sudo
-u
git sh
-c
'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \
git@newserver:/mnt/gitlab/repositories'
git@newserver:/mnt/gitlab/repositories'
```
```
## Thousands of Git repositories: use one
rsync
per repository
## Thousands of Git repositories: use one
`rsync`
per repository
Every time you start an
rsync
job it has to inspect all files in
Every time you start an
`rsync`
job it has to inspect all files in
the source directory, all files in the target directory, and then
the source directory, all files in the target directory, and then
decide what files to copy or not. If the source or target directory
decide what files to copy or not. If the source or target directory
has many contents this startup phase of
rsync
can become a burden
has many contents this startup phase of
`rsync`
can become a burden
for your GitLab server. In cases like this you can make
rsync
's
for your GitLab server. In cases like this you can make
`rsync`
's
life easier by dividing its work in smaller pieces, and sync one
life easier by dividing its work in smaller pieces, and sync one
repository at a time.
repository at a time.
In addition to
rsync
we will use
[
GNU
In addition to
`rsync`
we will use
[
GNU
Parallel
](
http://www.gnu.org/software/parallel/
)
. This utility is
Parallel
](
http://www.gnu.org/software/parallel/
)
. This utility is
not included in GitLab so you need to install it yourself with
apt
not included in GitLab so you need to install it yourself with
`apt`
or
yum
. Also note that the GitLab scripts we used below were added
or
`yum`
. Also note that the GitLab scripts we used below were added
in GitLab 8.1.
in GitLab 8.1.
**
This process does not clean up repositories at the target location that no
**
This process does not clean up repositories at the target location that no
...
@@ -90,9 +91,9 @@ longer exist at the source.** If you start using your GitLab instance with
...
@@ -90,9 +91,9 @@ longer exist at the source.** If you start using your GitLab instance with
`/mnt/gitlab/repositories`
, you need to run
`gitlab-rake gitlab:cleanup:repos`
`/mnt/gitlab/repositories`
, you need to run
`gitlab-rake gitlab:cleanup:repos`
after switching to the new repository storage directory.
after switching to the new repository storage directory.
### Parallel
rsync
for all repositories known to GitLab
### Parallel
`rsync`
for all repositories known to GitLab
This will sync repositories with 10
rsync
processes at a time. We keep
This will sync repositories with 10
`rsync`
processes at a time. We keep
track of progress so that the transfer can be restarted if necessary.
track of progress so that the transfer can be restarted if necessary.
First we create a new directory, owned by
`git`
, to hold transfer
First we create a new directory, owned by
`git`
, to hold transfer
...
@@ -147,7 +148,7 @@ cat /home/git/transfer-logs/* | sort | uniq -u |\
...
@@ -147,7 +148,7 @@ cat /home/git/transfer-logs/* | sort | uniq -u |\
`
`
```
```
### Parallel
rsync
only for repositories with recent activity
### Parallel
`rsync`
only for repositories with recent activity
Suppose you have already done one sync that started after 2015-10-1 12:00 UTC.
Suppose you have already done one sync that started after 2015-10-1 12:00 UTC.
Then you might only want to sync repositories that were changed via GitLab
Then you might only want to sync repositories that were changed via GitLab
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment