To increase visibility, we recommend that GitLab team members run feature flag
related Chatops commands within certain Slack channels based on the environment
and related feature. For the [staging](https://staging.gitlab.com)
and [development](https://dev.gitlab.org) environments of GitLab.com,
the commands should run in a channel for the stage the feature is relevant too.
For example, use the `#s_monitor` channel for features developed by the
Monitor stage, Health group.
For all production environment Chatops commands, use the `#production` channel.
As per the template, where a feature would have a (potentially) significant user
impact and the feature is being enabled instance wide prior to release, please copy
the Slack message and repost in the `#support_gitlab-com` channel for added visibility
and awareness, preferably with a link to the issue, MR, or docs.
Regardless of the channel in which the Chatops command is ran, any feature flag change that affects GitLab.com will automatically be logged in an issue.
The issue is created in the [gl-infra/feature-flag-log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/issues?scope=all&utf8=%E2%9C%93&state=closed) project, and it will at minimum log the Slack handle of person enabling a feature flag, the time, and the name of the flag being changed.
The issue is then also posted to GitLab Inc. internal [Grafana dashboard](https://dashboards.gitlab.net/) as an annotation marker to make the change even more visible.
Changes to the issue format can be submitted in the [Chatops project](https://gitlab.com/gitlab-com/chatops).
## Rolling out changes
When the changes are deployed to the environments it is time to start
...
...
@@ -67,19 +41,11 @@ If you get an error "Whoops! This action is not allowed. This incident
will be reported." that means your Slack account is not allowed to
change feature flags or you do not [have access](#access).
### Enabling feature for preproduction testing
### Enabling a feature for preproduction testing
As a first step in a feature rollout, you should enable the feature on <https://staging.gitlab.com>
and <https://dev.gitlab.org>.
For example, to enable a feature for 25% of all users, run the following in
Slack:
```shell
/chatops run feature set new_navigation_bar 25 --dev
/chatops run feature set new_navigation_bar 25 --staging
```
These two environments have different scopes.
`dev.gitlab.org` is a production CE environment that has internal GitLab Inc.
traffic and is used for some development and other related work.
...
...
@@ -89,13 +55,65 @@ a (very) rough estimate of how your feature will look/behave on GitLab.com.
Both of these instances are connected to Sentry so make sure you check the projects
there for any exceptions while testing your feature after enabling the feature flag.
Once you are confident enough that these environments are in a good state with your
feature enabled, you can roll out the change to GitLab.com.
For these preproduction environments, the commands should be run in a
Slack channel for the stage the feature is relevant to. For example, use the
`#s_monitor` channel for features developed by the Monitor stage, Health
group.
To enable a feature for 25% of all users, run the following in Slack:
```shell
/chatops run feature set new_navigation_bar 25 --dev
/chatops run feature set new_navigation_bar 25 --staging
```
### Enabling a feature for GitLab.com
Similar to above, to enable a feature for 25% of all users, run the following in
Slack:
When a feature has successfully been
[enabled on a preproduction](#enabling-a-feature-for-preproduction-testing)
environment and verified as safe and working, you can roll out the
change to GitLab.com (production).
#### Communicate the change
Some feature flag changes on GitLab.com should be communicated with
parts of the company. The developer responsible needs to determine
whether this is necessary and the appropriate level of communication.
This depends on the feature and what sort of impact it might have.
As a guideline:
- For simple features that are low-risk, and easily rolled back, then
just proceed to [enabling the feature in `#production`](#process).
- For features that will impact user experience consider notifying
`#support_gitlab-com` beforehand.
- For features with significant downstream effects (e.g.: turning on/off
Elasticsearch indexing) consider coordinating with `#production`
beforehand.
#### Process
Before toggling any feature flag, check that there are no ongoing
significant incidents on GitLab.com. You can do this by checking the
`#production` and `#incident-management` Slack channels, or looking for
