Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
625840ea
Commit
625840ea
authored
Apr 05, 2019
by
Jason Colyer
Committed by
Achilleas Pipinellis
Apr 05, 2019
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Update extra_sidekiq_processes.md to:
- Include doing it via gitlab.rb - Use full command path to avoid confusion
parent
bb9c3769
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
115 additions
and
27 deletions
+115
-27
doc/administration/operations/extra_sidekiq_processes.md
doc/administration/operations/extra_sidekiq_processes.md
+115
-27
doc/administration/operations/img/sidekiq-cluster.png
doc/administration/operations/img/sidekiq-cluster.png
+0
-0
No files found.
doc/administration/operations/extra_sidekiq_processes.md
View file @
625840ea
# Extra Sidekiq
P
rocesses
# Extra Sidekiq
p
rocesses
GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes
GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes
besides the default one. These processes can be used to consume a dedicated set
besides the default one. These processes can be used to consume a dedicated set
of queues. This can be used to ensure certain queues always have dedicated
of queues. This can be used to ensure certain queues always have dedicated
workers, no matter the
amount
of jobs that need to be processed.
workers, no matter the
number
of jobs that need to be processed.
## Starting Extra Processes
## Starting extra processes via Omnibus GitLab
To enable
`sidekiq-cluster`
, you must apply the
`sidekiq_cluster['enable'] = true`
setting
`/etc/gitlab/gitlab.rb`
:
```
ruby
sidekiq_cluster
[
'enable'
]
=
true
```
You will then specify how many additional processes to create via
`sidekiq-cluster`
as well as which queues for them to handle. This is done via the
`sidekiq_cluster['queue_groups']`
setting. This is an array whose items contain
which queues to process. Each item in the array will equate to one additional
sidekiq process.
As an example, to make additional sidekiq processes that process the
`elastic_indexer`
and
`mailers`
queues, you would apply the following:
```
ruby
sidekiq_cluster
[
'queue_groups'
]
=
[
"elastic_indexer"
,
"mailers"
]
```
To have an additional sidekiq process handle multiple queues, you simply put a
comma after the first queue name and then put the next queue name:
```
ruby
sidekiq_cluster
[
'queue_groups'
]
=
[
"elastic_indexer,elastic_commit_indexer"
,
"mailers"
]
```
Keep in mind, all changes must be followed by reconfiguring your GitLab
application via
`sudo gitlab-ctl reconfigure`
.
#### Monitoring
Once the Sidekiq processes are added, you can visit the "Background Jobs"
section under the admin area in GitLab (
`/admin/background_jobs`
).
![
Extra sidekiq processes
](
img/sidekiq-cluster.png
)
#### All queues with exceptions
To have the additional sidekiq processes work on every queue EXCEPT the ones
you list:
1.
Edit
`/etc/gitlab/gitlab.rb`
and add:
```
ruby
sidekiq_cluster
[
'negate'
]
=
true
```
1.
Save the file and
[
reconfigure GitLab
](
../restart_gitlab.md#omnibus-gitlab-reconfigure
)
for the changes to take effect.
#### Limiting concurrency
1.
Edit
`/etc/gitlab/gitlab.rb`
and add:
```
ruby
sidekiq_cluster
[
'concurrency'
]
=
25
```
1.
Save the file and
[
reconfigure GitLab
](
../restart_gitlab.md#omnibus-gitlab-reconfigure
)
for the changes to take effect.
Keep in mind, this normally would not exceed the number of CPU cores available.
#### Modifying the check interval
To modify the check interval for the additional Sidekiq processes:
1.
Edit
`/etc/gitlab/gitlab.rb`
and add:
```
ruby
sidekiq_cluster
[
'interval'
]
=
5
```
1.
Save the file and
[
reconfigure GitLab
](
../restart_gitlab.md#omnibus-gitlab-reconfigure
)
for the changes to take effect.
This tells the additional processes how often to check for enqueued jobs.
## Starting extra processes via command line
Starting extra Sidekiq processes can be done using the command
Starting extra Sidekiq processes can be done using the command
`bin/sidekiq-cluster`
. This command takes arguments using the following syntax:
`/opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster`
. This command
takes arguments using the following syntax:
```
bash
```
bash
sidekiq-cluster
[
QUEUE,QUEUE,...]
[
QUEUE, ...]
/opt/gitlab/embedded/service/gitlab-rails/bin/
sidekiq-cluster
[
QUEUE,QUEUE,...]
[
QUEUE, ...]
```
```
Each separate argument denotes a group of queues that have to be processed by a
Each separate argument denotes a group of queues that have to be processed by a
...
@@ -29,14 +115,14 @@ For example, say you want to start 2 extra processes: one to process the
...
@@ -29,14 +115,14 @@ For example, say you want to start 2 extra processes: one to process the
done as follows:
done as follows:
```
bash
```
bash
sidekiq-cluster process_commit post_receive
/opt/gitlab/embedded/service/gitlab-rails/bin/
sidekiq-cluster process_commit post_receive
```
```
If you instead want to start one process processing both queues you'd use the
If you instead want to start one process processing both queues you'd use the
following syntax:
following syntax:
```
bash
```
bash
sidekiq-cluster process_commit,post_receive
/opt/gitlab/embedded/service/gitlab-rails/bin/
sidekiq-cluster process_commit,post_receive
```
```
If you want to have one Sidekiq process process the "process_commit" and
If you want to have one Sidekiq process process the "process_commit" and
...
@@ -44,26 +130,20 @@ If you want to have one Sidekiq process process the "process_commit" and
...
@@ -44,26 +130,20 @@ If you want to have one Sidekiq process process the "process_commit" and
you'd use the following:
you'd use the following:
```
bash
```
bash
sidekiq-cluster process_commit,post_receive gitlab_shell
/opt/gitlab/embedded/service/gitlab-rails/bin/
sidekiq-cluster process_commit,post_receive gitlab_shell
```
```
## Concurrency
#### Monitoring
Each process started using
`sidekiq-cluster`
starts with a number of threads
that equals the number of queues, plus one spare thread. For example, a process
that processes "process_commit" and "post_receive" will use 3 threads in total.
## Monitoring
The
`sidekiq-cluster`
command will not terminate once it has started the desired
The
`sidekiq-cluster`
command will not terminate once it has started the desired
amount of Sidekiq processes. Instead the process will continue running and
amount of Sidekiq processes. Instead
,
the process will continue running and
forward any signals to the child processes. This makes it easy to stop all
forward any signals to the child processes. This makes it easy to stop all
Sidekiq processes as you simply send a signal to the
`sidekiq-cluster`
process,
Sidekiq processes as you simply send a signal to the
`sidekiq-cluster`
process,
instead of having to send it to the individual processes.
instead of having to send it to the individual processes.
If the
`sidekiq-cluster`
process crashes or
is SIGKILL'd the child processes
If the
`sidekiq-cluster`
process crashes or
receives a
`SIGKILL`
, the child
will terminate themselves after a few seconds. This ensures you don't end up
processes will terminate themselves after a few seconds. This ensures you don't
with zombie Sidekiq processes.
end up
with zombie Sidekiq processes.
All of this makes monitoring the processes fairly easy. Simply hook up
All of this makes monitoring the processes fairly easy. Simply hook up
`sidekiq-cluster`
to your supervisor of choice (e.g. runit) and you're good to
`sidekiq-cluster`
to your supervisor of choice (e.g. runit) and you're good to
...
@@ -75,26 +155,26 @@ process to terminate, then terminate itself. This removes the need for
...
@@ -75,26 +155,26 @@ process to terminate, then terminate itself. This removes the need for
Instead you should make sure your supervisor restarts the
`sidekiq-cluster`
Instead you should make sure your supervisor restarts the
`sidekiq-cluster`
process whenever necessary.
process whenever necessary.
##
PID F
iles
##
## PID f
iles
The
`sidekiq-cluster`
command can store its PID in a file. By default no PID
The
`sidekiq-cluster`
command can store its PID in a file. By default no PID
file is written, but this can be changed by passing the
`--pidfile`
option to
file is written, but this can be changed by passing the
`--pidfile`
option to
`sidekiq-cluster`
. For example:
`sidekiq-cluster`
. For example:
```
bash
```
bash
sidekiq-cluster
--pidfile
/var/run/gitlab/sidekiq_cluster.pid process_commit
/opt/gitlab/embedded/service/gitlab-rails/bin/
sidekiq-cluster
--pidfile
/var/run/gitlab/sidekiq_cluster.pid process_commit
```
```
Keep in mind that the PID file will contain the PID of the
`sidekiq-cluster`
Keep in mind that the PID file will contain the PID of the
`sidekiq-cluster`
command
,
and not the PID(s) of the started Sidekiq processes.
command and not the PID(s) of the started Sidekiq processes.
## Environment
##
##
Environment
The Rails environment can be set by passing the
`--environment`
flag to the
The Rails environment can be set by passing the
`--environment`
flag to the
`sidekiq-cluster`
command, or by setting
`RAILS_ENV`
to a non-empty value. The
`sidekiq-cluster`
command, or by setting
`RAILS_ENV`
to a non-empty value. The
default value is "development".
default value is "development".
##
All Queues With E
xceptions
##
## All queues with e
xceptions
You're able to run all queues in
`sidekiq_queues.yml`
file on a single or
You're able to run all queues in
`sidekiq_queues.yml`
file on a single or
multiple processes with exceptions using the
`--negate`
flag.
multiple processes with exceptions using the
`--negate`
flag.
...
@@ -112,14 +192,14 @@ For multiple processes of all queues (except "process_commit" and "post_receive"
...
@@ -112,14 +192,14 @@ For multiple processes of all queues (except "process_commit" and "post_receive"
sidekiq-cluster process_commit,post_receive process_commit,post_receive
--negate
sidekiq-cluster process_commit,post_receive process_commit,post_receive
--negate
```
```
##
Limiting C
oncurrency
##
## Limiting c
oncurrency
By default,
`sidekiq-cluster`
will spin up extra Sidekiq processes that use
By default,
`sidekiq-cluster`
will spin up extra Sidekiq processes that use
one thread per queue up to a maximum of 50. If you wish to change the cap, use
one thread per queue up to a maximum of 50. If you wish to change the cap, use
the
`-m N`
option. For example, this would cap the maximum number of threads to 1:
the
`-m N`
option. For example, this would cap the maximum number of threads to 1:
```
bash
```
bash
sidekiq-cluster process_commit,post_receive
-m
1
/opt/gitlab/embedded/service/gitlab-rails/bin/
sidekiq-cluster process_commit,post_receive
-m
1
```
```
For each queue group, the concurrency factor will be set to min(number of
For each queue group, the concurrency factor will be set to min(number of
...
@@ -127,4 +207,12 @@ queues, N). Setting the value to 0 will disable the limit.
...
@@ -127,4 +207,12 @@ queues, N). Setting the value to 0 will disable the limit.
Note that each thread requires a Redis connection, so adding threads may
Note that each thread requires a Redis connection, so adding threads may
increase Redis latency and potentially cause client timeouts. See the
[
Sidekiq
increase Redis latency and potentially cause client timeouts. See the
[
Sidekiq
documentation about Redis
](
https://github.com/mperham/sidekiq/wiki/Using-Redis
)
for more details.
documentation about Redis
](
https://github.com/mperham/sidekiq/wiki/Using-Redis
)
for more details.
## Number of threads
Each process started using
`sidekiq-cluster`
(whether it be via command line or
via the gitlab.rb file) starts with a number of threads that equals the number
of queues, plus one spare thread. For example, a process that handles the
"process_commit" and "post_receive" queues will use 3 threads in total.
doc/administration/operations/img/sidekiq-cluster.png
0 → 100644
View file @
625840ea
60.5 KB
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment