Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
1b008ad9
Commit
1b008ad9
authored
May 28, 2021
by
Achilleas Pipinellis
Committed by
Quang-Minh Nguyen
May 28, 2021
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Apply 6 suggestion(s) to 2 file(s)
parent
5373d661
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
16 additions
and
35 deletions
+16
-35
doc/administration/operations/extra_sidekiq_processes.md
doc/administration/operations/extra_sidekiq_processes.md
+1
-1
doc/administration/operations/extra_sidekiq_routing.md
doc/administration/operations/extra_sidekiq_routing.md
+15
-34
No files found.
doc/administration/operations/extra_sidekiq_processes.md
View file @
1b008ad9
...
@@ -117,7 +117,7 @@ you list:
...
@@ -117,7 +117,7 @@ you list:
> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
> - [Renamed from `experimental_queue_selector` to `queue_selector`](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab 13.6.
In addition to selecting queues by name, as above, the
`queue_selector`
option
In addition to selecting queues by name, as above, the
`queue_selector`
option
allows queue groups to be selected in a more general way using
[
worker matching
allows queue groups to be selected in a more general way using
a
[
worker matching
query
](
extra_sidekiq_routing.md#worker-matching-query
)
. After
`queue_selector`
query
](
extra_sidekiq_routing.md#worker-matching-query
)
. After
`queue_selector`
is set, all
`queue_groups`
must follow the aforementioned syntax.
is set, all
`queue_groups`
must follow the aforementioned syntax.
...
...
doc/administration/operations/extra_sidekiq_routing.md
View file @
1b008ad9
...
@@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
...
@@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w
# Queue routing rules **(FREE SELF)**
# Queue routing rules **(FREE SELF)**
When the number of Sidekiq jobs increases to a certain scale, the system faces
When the number of Sidekiq jobs increases to a certain scale, the system faces
some scalability issues. One of them is the length of the queue tends to get
some scalability issues. One of them is th
at th
e length of the queue tends to get
longer. High-urgency jobs have to wait longer until other less urgent jobs
longer. High-urgency jobs have to wait longer until other less urgent jobs
finish. This head-of-line blocking situation may eventually affect the
finish. This head-of-line blocking situation may eventually affect the
responsiveness of the system, especially critical actions. In another scenario,
responsiveness of the system, especially critical actions. In another scenario,
...
@@ -55,21 +55,21 @@ sidekiq['routing_rules'] = [
...
@@ -55,21 +55,21 @@ sidekiq['routing_rules'] = [
```
```
The routing rules list is an order-matter array of tuples of query and
The routing rules list is an order-matter array of tuples of query and
corresponding queue
.
corresponding queue
:
-
The query is following
[
worker matching query
](
#worker-matching-query
)
syntax.
-
The query is following a
[
worker matching query
](
#worker-matching-query
)
syntax.
-
The
`<queue_name>`
must be a valid Sidekiq queue name. If the queue name
-
The queue_name must be a valid Sidekiq queue name. If the queue name
is
`nil`
, or an empty string, the worker is routed to the queue generated
is nil, or an empty string, the worker is routed to the queue generated
by the name of the worker instead.
by the name of the worker instead.
The query supports wildcard matching
`*`
, which matches all workers. As a
The query supports wildcard matching
`*`
, which matches all workers. As a
result, the wildcard query must stay at the end of the list or the rules after it
result, the wildcard query must stay at the end of the list or the rules after it
are ignored.
are ignored.
> Important: Mixing queue routing rules and queue selectors requires care to
NOTE:
> ensure all jobs that are scheduled and picked up by appropriate Sidekiq
Mixing queue routing rules and queue selectors requires care to
> workers
ensure all jobs that are scheduled and picked up by appropriate Sidekiq
workers.
## Worker matching query
## Worker matching query
...
@@ -84,7 +84,7 @@ components:
...
@@ -84,7 +84,7 @@ components:
### Available attributes
### Available attributes
-
[
Introduced
](
https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261
)
in GitLab 13.1,
`tags`
.
> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1 (`tags`)
.
Queue matching query works upon the worker attributes, described in
[
Sidekiq
Queue matching query works upon the worker attributes, described in
[
Sidekiq
style guide
](
../../development/sidekiq_style_guide.md
)
. We support querying
style guide
](
../../development/sidekiq_style_guide.md
)
. We support querying
...
@@ -122,10 +122,10 @@ that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
...
@@ -122,10 +122,10 @@ that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have
neither of those tags.
neither of those tags.
The attributes of each worker are hard-coded in the source code. For
The attributes of each worker are hard-coded in the source code. For
convenience, we generate
[
list of all available attributes in
convenience, we generate
a
[
list of all available attributes in
FOSS
](
https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml
)
GitLab Community Edition
](
https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml
)
and
[
list of all available attributes in
and
a
[
list of all available attributes in
EE
](
https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml
)
.
GitLab Enterprise Edition
](
https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/all_queues.yml
)
.
### Available operators
### Available operators
...
@@ -155,25 +155,6 @@ have higher precedence than OR.
...
@@ -155,25 +155,6 @@ have higher precedence than OR.
later, as with the standard queue group syntax above, a single
`*`
as the
later, as with the standard queue group syntax above, a single
`*`
as the
entire queue group selects all queues.
entire queue group selects all queues.
### Example queries
```
plaintext
# Match workers having memory resource boundary
resource_boundary=memory
# Match low-urgency cpu-intensive workers
resource_boundary=cpu&urgency=default,low
# Match high-urgency cpu-intensive workers but not the ones require disk IO
resource_boundary=cpu&urgency=high&tags!=requires_disk_io
# Match throttled workers belong to database feature category
feature_category=database&urgency=throttled
# Match default and mailers queues
queue=default|queue=mailers
```
### Migration
### Migration
After the Sidekiq routing rules are changed, administrators need to take care
After the Sidekiq routing rules are changed, administrators need to take care
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment