chaos_endpoints.md 7.37 KB
Newer Older
1
# Generating chaos in a test GitLab instance
2 3 4

As [Werner Vogels](https://twitter.com/Werner), the CTO at Amazon Web Services, famously put it, **Everything fails, all the time**.

5
As a developer, it's as important to consider the failure modes in which your software will operate as much as normal operation. Doing so can mean the difference between a minor hiccup leading to a scattering of `500` errors experienced by a tiny fraction of users and a full site outage that affects all users for an extended period.
6 7 8

To paraphrase [Tolstoy](https://en.wikipedia.org/wiki/Anna_Karenina_principle), _all happy servers are alike, but all failing servers are failing in their own way_. Luckily, there are ways we can attempt to simulate these failure modes, and the chaos endpoints are tools for assisting in this process.

9
Currently, there are four endpoints for simulating the following conditions:
10

11 12 13 14
- Slow requests.
- CPU-bound requests.
- Memory leaks.
- Unexpected process crashes.
15

16 17
## Enabling chaos endpoints

18 19
For obvious reasons, these endpoints are not enabled by default on `production`.
They are enabled by default on **development** environments.
20

21
DANGER: **Danger:**
22 23
It is required that you secure access to the chaos endpoints using a secret token.
You should not enable them in production unless you absolutely know what you're doing.
24

25 26 27
A secret token can be set through the `GITLAB_CHAOS_SECRET` environment variable.
For example, when using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit)
this can be done with the following command:
28

29
```bash
30
GITLAB_CHAOS_SECRET=secret gdk run
31 32 33 34
```

Replace `secret` with your own secret token.

35
## Invoking chaos
36

37
Once you have enabled the chaos endpoints and restarted the application, you can start testing using the endpoints.
38

39 40 41 42
By default, when invoking a chaos endpoint, the web worker process which receives the request will handle it. This means, for example, that if the Kill
operation is invoked, the Puma or Unicorn worker process handling the request will be killed. To test these operations in Sidekiq, the `async` parameter on
each endpoint can be set to `true`. This will run the chaos process in a Sidekiq worker.

43
## Memory leaks
44 45 46

To simulate a memory leak in your application, use the `/-/chaos/leakmem` endpoint.

47 48
NOTE: **Note:**
The memory is not retained after the request finishes. Once the request has completed, the Ruby garbage collector will attempt to recover the memory.
49

50 51 52 53
```
GET /-/chaos/leakmem
GET /-/chaos/leakmem?memory_mb=1024
GET /-/chaos/leakmem?memory_mb=1024&duration_s=50
54
GET /-/chaos/leakmem?memory_mb=1024&duration_s=50&async=true
55 56
```

57 58 59
| Attribute    | Type    | Required | Description                                                                          |
| ------------ | ------- | -------- | ------------------------------------------------------------------------------------ |
| `memory_mb`  | integer | no       | How much memory, in MB, should be leaked. Defaults to 100MB.                         |
60
| `duration_s` | integer | no       | Minimum duration_s, in seconds, that the memory should be retained. Defaults to 30s. |
61
| `async`      | boolean | no       | Set to true to leak memory in a Sidekiq background worker process                    |
62

63 64
```bash
curl http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10 --header 'X-Chaos-Secret: secret'
65
curl http://localhost:3000/-/chaos/leakmem?memory_mb=1024&duration_s=10&token=secret
66
```
67

68
## CPU spin
69 70 71

This endpoint attempts to fully utilise a single core, at 100%, for the given period.

Evan Read's avatar
Evan Read committed
72
Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds).
73 74 75
If you're using Unicorn, this is done by killing the worker process.

```
76 77
GET /-/chaos/cpu_spin
GET /-/chaos/cpu_spin?duration_s=50
78
GET /-/chaos/cpu_spin?duration_s=50&async=true
79 80 81 82
```

| Attribute    | Type    | Required | Description                                                           |
| ------------ | ------- | -------- | --------------------------------------------------------------------- |
Evan Read's avatar
Evan Read committed
83
| `duration_s` | integer | no       | Duration, in seconds, that the core will be utilized. Defaults to 30s |
84
| `async`      | boolean | no       | Set to true to consume CPU in a Sidekiq background worker process     |
85 86 87 88 89 90 91 92 93 94 95

```bash
curl http://localhost:3000/-/chaos/cpu_spin?duration_s=60 --header 'X-Chaos-Secret: secret'
curl http://localhost:3000/-/chaos/cpu_spin?duration_s=60&token=secret
```

## DB spin

This endpoint attempts to fully utilise a single core, and interleave it with DB request, for the given period.
This endpoint can be used to model yielding execution to another threads when running concurrently.

Evan Read's avatar
Evan Read committed
96
Depending on your rack server setup, your request may timeout after a predetermined period (normally 60 seconds).
97 98 99 100 101
If you're using Unicorn, this is done by killing the worker process.

```
GET /-/chaos/db_spin
GET /-/chaos/db_spin?duration_s=50
102
GET /-/chaos/db_spin?duration_s=50&async=true
103 104
```

105 106 107
| Attribute    | Type    | Required | Description                                                                 |
| ------------ | ------- | -------- | --------------------------------------------------------------------------- |
| `interval_s` | float   | no       | Interval, in seconds, for every DB request. Defaults to 1s                  |
Evan Read's avatar
Evan Read committed
108
| `duration_s` | integer | no       | Duration, in seconds, that the core will be utilized. Defaults to 30s       |
109
| `async`      | boolean | no       | Set to true to perform the operation in a Sidekiq background worker process |
110 111

```bash
112 113
curl http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60 --header 'X-Chaos-Secret: secret'
curl http://localhost:3000/-/chaos/db_spin?interval_s=1&duration_s=60&token=secret
114 115
```

116
## Sleep
117

118
This endpoint is similar to the CPU Spin endpoint but simulates off-processor activity, such as network calls to backend services. It will sleep for a given duration_s.
119

120
As with the CPU Spin endpoint, this may lead to your request timing out if duration_s exceeds the configured limit.
121

122 123 124
```
GET /-/chaos/sleep
GET /-/chaos/sleep?duration_s=50
125
GET /-/chaos/sleep?duration_s=50&async=true
126
```
127

128 129 130
| Attribute    | Type    | Required | Description                                                            |
| ------------ | ------- | -------- | ---------------------------------------------------------------------- |
| `duration_s` | integer | no       | Duration, in seconds, that the request will sleep for. Defaults to 30s |
131
| `async`      | boolean | no       | Set to true to sleep in a Sidekiq background worker process            |
132 133

```bash
134
curl http://localhost:3000/-/chaos/sleep?duration_s=60 --header 'X-Chaos-Secret: secret'
135
curl http://localhost:3000/-/chaos/sleep?duration_s=60&token=secret
136 137
```

138
## Kill
139

140
This endpoint will simulate the unexpected death of a worker process using a `kill` signal.
141

142 143
NOTE: **Note:**
Since this endpoint uses the `KILL` signal, the worker is not given a chance to cleanup or shutdown.
144

145 146
```
GET /-/chaos/kill
147
GET /-/chaos/kill?async=true
148
```
149

150 151 152 153
| Attribute    | Type    | Required | Description                                                            |
| ------------ | ------- | -------- | ---------------------------------------------------------------------- |
| `async`      | boolean | no       | Set to true to kill a Sidekiq background worker process                |

154
```bash
155
curl http://localhost:3000/-/chaos/kill --header 'X-Chaos-Secret: secret'
156
curl http://localhost:3000/-/chaos/kill?token=secret
157
```