Merge branch 'sy-incident-escalations-documentation' into 'master'

Add documentation for incident escalations

See merge request gitlab-org/gitlab!80441

doc/operations/incident_management/alerts.md

@@ -144,6 +144,7 @@ The following actions result in a system note:
 - [Updating the status of an alert](#update-an-alerts-status)
 - [Creating an incident based on an alert](#create-an-incident-from-an-alert)
 - [Assignment of an alert to a user](#assign-an-alert)
+- [Escalation of an alert to on-call responders](paging.md#escalating-an-alert)
 
 ![Alert Details Activity Feed](img/alert_detail_activity_feed_v13_5.png)
 
@@ -153,8 +154,26 @@ There are different actions available in GitLab to help triage and respond to al
 
 ### Update an alert's status
 
-The Alert detail view enables you to update the Alert Status.
-See [Create and manage alerts in GitLab](alerts.md) for more details.
+**Triggered** is the default status for new alerts. For users with the Developer role or higher, the
+alert status can be updated from these locations:
+
+- [Alert list](#alert-list): select the status dropdown corresponding to an alert, then select an
+  alternate status.
+- [Alert details page](#alert-details-page): select **Edit** in the right-hand side bar, then select
+  an alternate status.
+
+To stop email notifications for alert reoccurrences in projects with [email notifications enabled](paging.md#email-notifications-for-alerts),
+[change the alert's status](alerts.md#update-an-alerts-status) away from **Triggered**.
+
+In projects with GitLab Premium, on-call responders can respond to [alert pages](paging.md#escalating-an-alert)
+by changing the status. Setting the status to:
+
+- **Resolved** silences all on-call pages for the alert.
+- **Acknowledged** limits on-call pages based on the project's [escalation policy](escalation_policies.md).
+- **Triggered** from **Resolved** restarts the alert escalating from the beginning.
+
+For [alerts with an associated incident](alerts.md#create-an-incident-from-an-alert),
+updating the alert status also updates the incident status.
 
 ### Create an incident from an alert
 
@@ -165,8 +184,10 @@ description populated from an alert. To create the issue,
 select the **Create Issue** button. You can then view the issue from the
 alert by selecting the **View Issue** button.
 
-Closing a GitLab issue associated with an alert changes the alert's status to
-Resolved. See [Create and manage alerts in GitLab](alerts.md) for more details
+You can also [create incidents for alerts automatically](incidents.md#create-incidents-automatically).
+
+Closing a GitLab issue associated with an alert [changes the alert's status](#update-an-alerts-status) to
+**Resolved**. See [Alert List](#alert-list) for more details
 about alert statuses.
 
 ### Assign an alert

doc/operations/incident_management/incidents.md

@@ -54,7 +54,7 @@ GitLab to create incident automatically whenever an alert is triggered:
 1. Check the **Create an incident** checkbox.
 1. To customize the incident, select an
    [issue template](../../user/project/description_templates.md#create-an-issue-template).
-1. To send [an email notification](paging.md#email-notifications) to users
+1. To send [an email notification](paging.md#email-notifications-for-alerts) to users
    with the Developer role, select
    **Send a separate email notification to Developers**. Email notifications are
    also sent to users with the **Maintainer** and **Owner** roles.
@@ -89,9 +89,9 @@ For users with at least Guest [permissions](../../user/permissions.md), the
 Incident list is available at **Monitor > Incidents**
 in your project's sidebar. The list contains the following metrics:
 
-![Incident List](img/incident_list_v13_5.png)
+![Incident List](img/incident_list_v14_9.png)
 
-- **Status** - To filter incidents by their status, click **Open**, **Closed**,
+- **State** - To filter incidents by their state, select **Open**, **Closed**,
   or **All** above the incident list.
 - **Search** - The Incident list supports a simple free text search, which filters
   on the **Title** and **Incident** fields.
@@ -108,6 +108,13 @@ in your project's sidebar. The list contains the following metrics:
 
 - **Incident** - The description of the incident, which attempts to capture the
   most meaningful data.
+- **Status** - The status of the incident, which can be one of the following values:
+  - **Triggered**
+  - **Acknowledged**
+  - **Resolved**
+
+  In GitLab Premium, this field is also linked to [on-call escalation](paging.md#escalating-an-incident) for the incident.
+
 - **Date created** - How long ago the incident was created. This field uses the
   standard GitLab pattern of `X time ago`, but is supported by a granular date/time
   tooltip depending on the user's locale.
@@ -246,6 +253,40 @@ You can also change the severity using the [`/severity` quick action](../../user
 Add a to-do for incidents that you want to track in your to-do list. Click the
 **Add a to do** button at the top of the right-hand side bar to add a to-do item.
 
+### Change incident status
+
+> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9.
+
+For users with the Developer role or higher, select **Edit** in the **Status** section of the
+right-hand side bar of an incident, then select a status. **Triggered** is the default status for
+new incidents.
+
+In projects with GitLab Premium, on-call responders can respond to [incident pages](paging.md#escalating-an-incident)
+by changing the status. Setting the status to:
+
+- **Resolved** silences on-call pages for the alert.
+- **Acknowledged** limits on-call pages based on the selected [escalation policy](#change-escalation-policy).
+- **Triggered** from **Resolved** restarts the incident escalating from the beginning.
+
+For [incidents created from alerts](alerts.md#create-an-incident-from-an-alert),
+updating the incident status also updates the alert status.
+
+## Change escalation policy **(PREMIUM)**
+
+> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9.
+
+For users with the Developer role or higher, select **Edit** in the **Escalation policy** section of
+the right-hand side bar of an incident, then select a policy. By default, new incidents do not have
+an escalation policy selected.
+
+Selecting an escalation policy updates the incident status to **Triggered** and begins
+[escalating the incident to on-call responders](paging.md#escalating-an-incident).
+Deselecting an escalation policy halts escalation. Refer to the [incident status](#change-incident-status)
+to manage on-call paging once escalation has begun.
+
+For [incidents created from alerts](alerts.md#create-an-incident-from-an-alert),
+the incident's escalation policy reflects the alert's escalation policy and cannot be changed.
+
 ### Manage incidents from Slack
 
 Slack slash commands allow you to control GitLab and view GitLab content without leaving Slack.

doc/operations/incident_management/oncall_schedules.md

@@ -10,8 +10,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 
 Use on-call schedule management to create schedules for responders to rotate on-call
 responsibilities. Maintain the availability of your software services by putting your teams on-call.
-With an on-call schedule, your team is notified immediately when things go wrong so they can quickly
-respond to service outages and disruptions.
+With [escalation policies](escalation_policies.md) and on-call schedules, your team is notified immediately
+when things go wrong so they can quickly respond to service outages and disruptions.
 
 To use on-call schedules:
 
@@ -111,9 +111,7 @@ Hover over any rotation shift participants in the schedule to view their individ
 
 ## Page an on-call responder
 
-When an alert is created in a project, GitLab sends an email to the on-call responder(s) in the
-on-call schedule for that project. If there is no schedule or no one on-call in that schedule at the
-time the alert is triggered, no email is sent.
+See [Paging](paging.md#paging) for more details.
 
 ## Removal or deletion of on-call user
 

doc/operations/incident_management/paging.md

@@ -21,7 +21,7 @@ receive a **single** page via Slack. To set up Slack notifications on your mobil
 device, make sure to enable notifications for the Slack app on your phone so
 you never miss a page.
 
-## Email notifications
+## Email notifications for alerts
 
 Email notifications are available in projects for triggered alerts. Project
 members with the **Owner** or **Maintainer** roles have the option to receive
@@ -34,8 +34,30 @@ a single email notification for new alerts.
    **Send a single email notification to Owners and Maintainers for new alerts** checkbox.
 1. Select **Save changes**.
 
+[Update the alert's status](alerts.md#update-an-alerts-status) to manage email notifications for an alert.
+
 ## Paging **(PREMIUM)**
 
-In projects that have an [on-call schedule](oncall_schedules.md) configured, on-call responders are
-paged through email for triggered alerts. The on-call responder(s) receive one email for triggered
-alerts.
+In projects that have an [escalation policy](escalation_policies.md) configured, on-call responder(s)
+can be automatically paged about critical problems through email.
+
+### Escalating an alert
+
+When an alert is triggered, it begins escalating to the on-call responders immediately.
+For each escalation rule in the project's escalation policy, the designated on-call
+responders receive one email when the rule fires. You can respond to a page
+or stop alert escalations by [updating the alert's status](alerts.md#update-an-alerts-status).
+
+### Escalating an incident
+
+> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5716) in GitLab 14.9.
+
+For incidents, paging on-call responders is optional for each individual incident.
+To begin escalating the incident, [set the incident's escalation policy](incidents.md#change-escalation-policy).
+For each escalation rule, the designated on-call responders receive one email when
+the rule fires. You can respond to a page or stop incident escalations by
+[updating the incident's status](incidents.md#change-incident-status) or, if applicable,
+[unsetting the incident's escalation policy](incidents.md#change-escalation-policy).
+
+To avoid duplicate pages, [incidents created from alerts](alerts.md#create-an-incident-from-an-alert) do not support independent escalation.
+Instead, the status and escalation policy fields are synced between the alert and the incident.

doc/user/project/integrations/webhook_events.md

@@ -202,6 +202,10 @@ The available values for `object_attributes.action` in the payload are:
 The `assignee` and `assignee_id` keys are deprecated
 and contain the first assignee only.
 
+The `escalation_status` and `escalation_policy` fields are
+only available for issue types which support escalations,
+such as incidents.
+
 Request header:
 
 ```plaintext
@@ -271,6 +275,12 @@ Payload example:
     "url": "http://example.com/diaspora/issues/23",
     "state": "opened",
     "action": "open",
+    "severity": "high",
+    "escalation_status": "triggered",
+    "escalation_policy": {
+      "id": 18,
+      "name": "Engineering On-call"
+    },
     "labels": [{
         "id": 206,
         "title": "API",

doc/user/project/settings/index.md

@@ -541,7 +541,7 @@ Configure [alert integrations](../../../operations/incident_management/integrati
 
 #### Alert integration
 
-Automatically [create](../../../operations/incident_management/incidents.md#create-incidents-automatically), [notify on](../../../operations/incident_management/paging.md#email-notifications), and [resolve](../../../operations/incident_management/incidents.md#automatically-close-incidents-via-recovery-alerts) incidents based on GitLab alerts.
+Automatically [create](../../../operations/incident_management/incidents.md#create-incidents-automatically), [notify on](../../../operations/incident_management/paging.md#email-notifications-for-alerts), and [resolve](../../../operations/incident_management/incidents.md#automatically-close-incidents-via-recovery-alerts) incidents based on GitLab alerts.
 
 #### PagerDuty integration