Commit 1950df2a authored by Phil Calder's avatar Phil Calder

Update experiment guide doc

parent 24037b7c
# Experiment Guide # Experiment Guide
Experiments will be conducted by teams from the [Growth Section](https://about.gitlab.com/handbook/engineering/development/growth/) and are not tied to releases, because they will primarily target GitLab.com. Experiments can be conducted by any GitLab team, most often the teams from the [Growth Sub-department](https://about.gitlab.com/handbook/engineering/development/growth/). Experiments are not tied to releases because they will primarily target GitLab.com.
Experiments will be run as an A/B test and will be behind a feature flag to turn the test on or off. Based on the data the experiment generates, the team will decide if the experiment had a positive impact and will be the new default or rolled back. Experiments will be run as an A/B test and will be behind a feature flag to turn the test on or off. Based on the data the experiment generates, the team will decide if the experiment had a positive impact and will be the new default or rolled back.
## Follow-up issue ## Experiment tracking issue
Each experiment requires a follow-up issue to resolve the experiment. Immediately after an experiment is deployed, the deadline of the issue needs to be set (this depends on the experiment but can be up to a few weeks in the future). Each experiment should have an [Experiment tracking](https://gitlab.com/groups/gitlab-org/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=growth%20experiment&search=%22Experiment+tracking%22) issue to track the experiment from roll-out through to cleanup/removal. Immediately after an experiment is deployed, the due date of the issue should be set (this depends on the experiment but can be up to a few weeks in the future).
After the deadline, the issue needs to be resolved and either: After the deadline, the issue needs to be resolved and either:
- It was successful and the experiment will be the new default. - It was successful and the experiment will be the new default.
...@@ -17,10 +17,10 @@ In either case, an outcome of the experiment should be posted to the issue with ...@@ -17,10 +17,10 @@ In either case, an outcome of the experiment should be posted to the issue with
## Code reviews ## Code reviews
Since the code of experiments will not be part of the codebase for a long time and we want to iterate fast to retrieve data,the code quality of experiments might sometimes not fulfill our standards but should not negatively impact the availability of GitLab whether the experiment is running or not. Since the code of experiments will not be part of the codebase for a long time and we want to iterate fast to retrieve data,the code quality of experiments might sometimes not fulfill our standards but should not negatively impact the availability of GitLab whether the experiment is running or not.
Experiments will only be deployed to a fraction of users but we still want a flawless experience for those users. Therefore, experiments still require tests. Initially experiments will only be deployed to a fraction of users but we still want a flawless experience for those users. Therefore, experiments still require tests.
For reviewers and maintainers: if you find code that would usually not make it through the review, but is temporarily acceptable, please mention your concerns but note that it's not necessary to change. For reviewers and maintainers: if you find code that would usually not make it through the review, but is temporarily acceptable, please mention your concerns but note that it's not necessary to change.
The author then adds a comment to this piece of code and adds a link to the issue that resolves the experiment. The author then adds a comment to this piece of code and adds a link to the issue that resolves the experiment. If the experiment is successful and becomes part of the product these follow up issues should be addressed.
## How to create an A/B test ## How to create an A/B test
...@@ -42,7 +42,7 @@ The author then adds a comment to this piece of code and adds a link to the issu ...@@ -42,7 +42,7 @@ The author then adds a comment to this piece of code and adds a link to the issu
- Use the experiment in a controller: - Use the experiment in a controller:
```ruby ```ruby
class RegistrationController < Applicationcontroller class RegistrationController < ApplicationController
def show def show
# experiment_enabled?(:feature_name) is also available in views and helpers # experiment_enabled?(:feature_name) is also available in views and helpers
if experiment_enabled?(:signup_flow) if experiment_enabled?(:signup_flow)
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment