Merge branch 'growth-experiment-guide-update' into 'master'

Growth experiment guide and rollout template update

See merge request gitlab-org/gitlab!69335

.gitlab/issue_templates/experiment_tracking_template.md → .gitlab/issue_templates/Experiment Rollout.md

-<!-- Title suggestion: [Experiment Tracking] experiment-key - description of experiment -->
+<!-- Title suggestion: [Experiment Rollout] experiment-key - description of experiment -->
 
-## What
+## Summary
 
-Track the status of an experiment through to removal.
+This issue tracks the rollout and status of an experiment through to removal.
 
-1. Experiment key: `<experiment-key>`
-1. Framework: `experimentation.rb` | `gitlab_experiment`
-1. Feature flag name: <experiment-key>_experiment_percentage` | `<experiment-key>`
-
-This is an experiment tracking issue for: `<issue or epic link>` 
-using the scoped [experiment label](https://about.gitlab.com/handbook/engineering/development/growth/#experiment-tracking-issue).
+1. Experiment key / feature flag name: `<experiment-key>`
+1. Epic or issue link: `<issue or epic link>`
 
+This is an experiment rollout issue
+using the scoped [experiment label](https://about.gitlab.com/handbook/engineering/development/growth/experimentation/#experiment-rollout-issue). 
 As well as defining the experiment rollout and cleanup, this issue incorporates the relevant 
 [`Feature Flag Roll Out`](https://gitlab.com/gitlab-org/gitlab/-/edit/master/.gitlab/issue_templates/Feature%20Flag%20Roll%20Out.md) steps. 
 
@@ -19,49 +17,65 @@ As well as defining the experiment rollout and cleanup, this issue incorporates
 - Team: `group::TEAM_NAME`
 - Most appropriate slack channel to reach out to: `#g_TEAM_NAME`
 - Best individual to reach out to: NAME
+- Product manager (PM): NAME
+
+### Stakeholders
+
+<!--
+Are there any other stages or teams involved that need to be kept in the loop?
+
+- PM: Name
+- Group: `group::TEAM_NAME`
+- The Support Team
+- The Delivery Team
+-->
 
 ## Expectations
 
 ### What are we expecting to happen?
 
+<!-- Describe the expected outcome when rolling out this experiment. -->
+
 ### What might happen if this goes wrong?
 
+<!-- Any MRs that need to be rolled back? Communication that needs to happen? What are some things you can think of that could go wrong - data loss or broken pages? -->
+
 ### What can we monitor to detect problems with this?
-<!-- Which dashboards from https://dashboards.gitlab.net are most relevant? Sentry errors reports can also be useful to review -->
 
-### Tracked data
+<!-- Which dashboards from https://dashboards.gitlab.net are most relevant? -->
+
+## Tracked data
 <!-- brief description or link to issue or Sisense dashboard -->
 
- Note: you can utilize [CXL calculator](https://cxl.com/ab-test-calculator/) to determine if your experiment has reached signifigance, it also includes an estimate for how much longer an experiment will need to run for before reaching signifigance.
+Note: you can use the [CXL calculator](https://cxl.com/ab-test-calculator/) to determine if your experiment has reached significance. The calculator includes an estimate for how much longer an experiment must run for before reaching significance.
 
-### Staging Test
-<!-- For experiments using `experimentation.rb`: To force this experiment on staging use `?force_experiment=<experiment-key>` -->
-<!-- list any steps required to setup this experiment, and link to a separate Staging environment test issue is applicable -->
+## Rollout plan
+<!-- Add an overview and method for modifying the feature flag -->
 
-<!-- uncomment if testing with specific groups/projects on GitLab.com
-## Beta groups/projects
+- Runtime in days, or until we expect to reach statistical significance: `30`
+- We will roll this out behind a feature flag and expose this to `<rollout-percentage>`% of actors to start then ramp it up from there.
 
-If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.
+`/chatops run feature set <experiment-key> <rollout-percentage> --actors`
 
-- `gitlab-org/gitlab` project
-- `gitlab-org`/`gitlab-com` groups
-- ...
--->
+### Status
 
-### Experiment tracking log
-<!-- Add an overview and method for modifying the feature flag
 
-* Runtime: 30 days or until we reach statistical significance
-* We will roll this out behind a feature flag and expose this to 20% of users to start then ramp it up from there.
-* feature flag based on experiment key `<experiment-key>` (see `experimentation.rb` in GitLab, append '_experiment_percentage')
+#### Preferred workflow
 
-`/chatops run feature set <experiment-key>_experiment_percentage <INITIAL_PERCENTAGE>`
--->
-<!-- Add bullet points to track changes to the rollout of this experiment (feature flag changes) 
+The issue should be assigned to the Product manager (PM) or Engineer (Eng) as follows:
 
-* YYYY-MM-DD UTC - initial rollout to 20% of users
-* TBD - review - increase to 50% of users
--->
+1. PM determines and manages the status of the experiment (assign this issue to the PM)
+1. PM asks for initial rollout on production, or changes to the status (assign to an Eng)
+1. Eng changes the status using `chatops` (reassign to the PM)
+1. When concluded, PM updates the 'Roll Out Steps' and adds a milestone (assigns to an Eng)
+
+The current status and history can be viewed using the: 
+
+- [API](https://gitlab.com/api/v4/experiments) (GitLab team members)
+- [Feature flag log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues?scope=all&utf8=%E2%9C%93&state=all) (GitLab team members)
+- [Experiment rollout board](https://gitlab.com/groups/gitlab-org/-/boards/1352542)
+
+In this rollout issue, ensure the scoped `experiment::` label is kept accurate.
 
 ### Experiment Results
 <!-- update when experiment in/validated, set the scoped `~experiment::` status accordingly -->
@@ -69,7 +83,7 @@ If applicable, any groups/projects that are happy to have this feature turned on
 ## Roll Out Steps
 
 - [ ] Confirm that QA tests pass with the feature flag enabled (if you're unsure how, contact the relevant [stable counterpart in the Quality department](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors))
-- [ ] Enable on staging (`/chatops run feature set feature_name true --staging`)
+- [ ] Enable on staging (`/chatops run feature set <experiment-key> true --staging`)
 - [ ] Test on staging
 - [ ] Ensure that documentation has been updated
 - [ ] Enable on GitLab.com for individual groups/projects listed above and verify behaviour  (`/chatops run feature set --project=gitlab-org/gitlab feature_name true`)
@@ -88,8 +102,8 @@ If applicable, any groups/projects that are happy to have this feature turned on
 - [ ] This feature can be disabled by running the following Chatops command:
 
 ```
-/chatops run feature set feature_name false
+/chatops run feature set <experiment-key> false
 ```
 
-/label ~"feature flag" ~"devops::growth" ~"growth experiment" ~"experiment tracking" ~Engineering ~"workflow::scheduling" ~"experiment::pending"
-
+/label ~"feature flag" ~"devops::growth" ~"growth experiment" ~"experiment-rollout" ~Engineering ~"workflow::scheduling" ~"experiment::pending"
+/milestone %"Next 1-3 releases" 

doc/development/experiment_guide/index.md

@@ -8,11 +8,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 
 Experiments can be conducted by any GitLab team, most often the teams from the [Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/). Experiments are not tied to releases because they primarily target GitLab.com.
 
-Experiments are run as an A/B/n test, and are behind a feature flag to turn the test on or off. Based on the data the experiment generates, the team decides if the experiment had a positive impact and should be made the new default, or rolled back.
+Experiments are run as an A/B/n test, and are behind an [experiment feature flag](../feature_flags/#experiment-type) to turn the test on or off. Based on the data the experiment generates, the team decides if the experiment had a positive impact and should be made the new default, or rolled back.
 
-## Experiment tracking issue
+## Experiment rollout issue
 
-Each experiment should have an [Experiment tracking](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&state=opened&label_name[]=growth%20experiment&search=%22Experiment+tracking%22) issue to track the experiment from roll-out through to cleanup/removal. The tracking issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. Immediately after an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future).
+Each experiment should have an [experiment rollout](https://gitlab.com/groups/gitlab-org/-/boards/1352542) issue to track the experiment from rollout through to cleanup and removal. 
+The rollout issue is similar to a feature flag rollout issue, and is also used to track the status of an experiment. 
+When an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future).
 After the deadline, the issue needs to be resolved and either:
 
 - It was successful and the experiment becomes the new default.
@@ -38,22 +40,14 @@ addressed.
 
 ## Implementing an experiment
 
-There are currently two options when implementing an experiment.
+[`GLEX`](https://gitlab.com/gitlab-org/gitlab-experiment) - or `Gitlab::Experiment`, the `gitlab-experiment` gem - is the preferred option for implementing an experiment in GitLab.
 
-One is built into GitLab directly and has been around for a while (this is called
-`Exerimentation Module`), and the other is provided by
-[`gitlab-experiment`](https://gitlab.com/gitlab-org/gitlab-experiment) and is referred
-to as `Gitlab::Experiment` -- GLEX for short.
+For more information, see [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md).
 
-Both approaches use [experiment](../feature_flags/index.md#experiment-type)
-feature flags. We recommend using GLEX rather than `Experimentation Module` for new experiments.
+There are still some longer running experiments using the [`Exerimentation Module`](experimentation.md).
 
-- [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md)
-- [Implementing an A/B experiment using `Experimentation Module`](experimentation.md)
-
-Historical Context: `Experimentation Module` was built iteratively with the needs that
-appeared while implementing Growth sub-department experiments, while GLEX was built
-with the findings of the team and an easier to use API.
+Both approaches use [experiment](../feature_flags/index.md#experiment-type) feature flags. 
+`GLEX` is the preferred option for new experiments.
 
 ### Add new icons and illustrations for experiments