Commit bbf3bdb4 authored by Ben Prescott @bprescott_↙ ☺'s avatar Ben Prescott @bprescott_↙ ☺ Committed by Marcia Ramos

docs: tune puma worker memory

parent ae498f68
...@@ -36,6 +36,14 @@ For more details about the Puma configuration, see the ...@@ -36,6 +36,14 @@ For more details about the Puma configuration, see the
## Puma Worker Killer ## Puma Worker Killer
Puma forks worker processes as part of a strategy to reduce memory use.
Each time a worker is created, it shares memory with the primary process and
only uses additional memory when it makes changes or additions to its memory pages.
Memory use by workers therefore increases over time, and Puma Worker Killer is the
mechanism that recovers this memory.
By default: By default:
- The [Puma Worker Killer](https://github.com/schneems/puma_worker_killer) restarts a worker if it - The [Puma Worker Killer](https://github.com/schneems/puma_worker_killer) restarts a worker if it
...@@ -56,6 +64,47 @@ To change the memory limit setting: ...@@ -56,6 +64,47 @@ To change the memory limit setting:
sudo gitlab-ctl reconfigure sudo gitlab-ctl reconfigure
``` ```
There are costs associated with killing and replacing workers including
reduced capacity to run GitLab, and CPU that is consumed
restarting the workers. `per_worker_max_memory_mb` should be set to a
higher value if the worker killer is replacing workers too often.
Worker count is calculated based on CPU cores, so a small GitLab deployment
wih 4-8 workers may experience performance issues if workers are being restarted
frequently, once or more per minute. This is too often.
A higher value of `1200` or more would be beneficial if the server has free memory.
The worker killer checks every 20 seconds, and can be monitored using
[the Puma log](../logs.md#puma_stdoutlog) `/var/log/gitlab/puma/puma_stdout.log`.
For example, for GitLab 13.5:
```plaintext
PumaWorkerKiller: Out of memory. 4 workers consuming total: 4871.23828125 MB
out of max: 4798.08 MB. Sending TERM to pid 26668 consuming 1001.00390625 MB.
```
From this output:
- The formula that calculates the maximum memory value results in workers
being killed before they reach the `per_worker_max_memory_mb` value.
- The default values for the formula before GitLab 13.5 were 550MB for the primary
and `per_worker_max_memory_mb` specified 850MB for each worker.
- As of GitLab 13.5 the values are primary: 800MB, worker: 1024MB.
- The threshold for workers to be killed is set at 98% of the limit:
```plaintext
0.98 * ( 800 + ( worker_processes * 1024MB ) )
```
- In the log output above, `0.98 * ( 800 + ( 4 * 1024 ) )` returns the
`max: 4798.08 MB` value.
Increasing the maximum to `1200`, for example, would set a `max: 5488 MB` value.
Workers use additional memory on top of the shared memory, how much
depends on a site's use of GitLab.
## Worker timeout ## Worker timeout
A [timeout of 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/rack_timeout.rb) A [timeout of 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/rack_timeout.rb)
...@@ -95,7 +144,6 @@ considered as a fair tradeoff in a memory-constraint environment. ...@@ -95,7 +144,6 @@ considered as a fair tradeoff in a memory-constraint environment.
When running Puma in Single mode, some features are not supported: When running Puma in Single mode, some features are not supported:
- Phased restart do not work: [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/300665)
- [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) - [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665)
- [Puma Worker Killer](https://gitlab.com/gitlab-org/gitlab/-/issues/300664) - [Puma Worker Killer](https://gitlab.com/gitlab-org/gitlab/-/issues/300664)
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment