At GitLab, we collect telemetry for the purpose of helping us build a better GitLab. Data about how GitLab is used is collected to better understand what parts of GitLab needs improvement and what features to build next. Telemetry also helps our team better understand the reasons why people use GitLab and with this knowledge we are able to make better product decisions.
We also encourage users to enable tracking, and we embrace full transparency with our tracking approach so it can be easily understood and trusted. By enabling tracking, users can:
- Contribute back to the wider community.
- Help GitLab improve on the product.
This documentation consists of three guides providing an overview of Telemetry at GitLab.
Telemetry Guide:
1.[Our tracking tools](#our-tracking-tools)
1.[What data can be tracked](#what-data-can-be-tracked)
1.[Telemetry systems overview](#telemetry-systems-overview)
[Usage Ping Guide](usage_ping.md)
1.[What is Usage Ping](usage_ping.md#what-is-usage-ping)
In this section we will explain the six different technologies we use to gather product usage data.
**Snowplow JS (Frontend)**
Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. [Snowplow JS](https://github.com/snowplow/snowplow/wiki/javascript-tracker) is a frontend tracker for client-side events.
**Snowplow Ruby (Backend)**
Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. [Snowplow Ruby](https://github.com/snowplow/snowplow/wiki/ruby-tracker) is a backend tracker for server-side events.
**Usage Ping**
Usage Ping is a method for GitLab Inc to collect usage data on a GitLab instance. Usage Ping is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. This high-level data is used to help our product, support, and sales teams.
Read more about how this works in the [Usage Ping guide](usage_ping.md)
**Database import**
Database imports are full imports of data into GitLab's data warehouse. For GitLab.com, the PostgreSQL database is loaded into Snowflake data warehouse every 6 hours. For more details, see the [data team handbook](https://about.gitlab.com/handbook/business-ops/data-team/#extract-and-load).
**Log system**
System logs are the application logs generated from running the GitLab Rails application. For more details, see the [log system](../../administration/logs.md) and [logging infrastructure](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#logging-infrastructure-overview).
## What data can be tracked
Our different tracking tools allows us to track different types of events. The event types and examples of what data can be tracked are outlined below.
- How many Projects have been created by unique users
- How many users logged in the past 28 day
Database counts are row counts for different tables in an instance’s database. These are SQL count queries which have been filtered, grouped, or aggregated which provide high level usage data. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql)
**Pageview events**
- How many sessions visited the /dashboard/groups page
**UI Events**
- How many sessions clicked on a button or link
- How many sessions closed a modal
UI events are any interface-driven actions from the browser including click data.
**CRUD or API events**
- How many Git pushes were made
- How many GraphQL queries were made
- How many requests were made to a Rails action or controller.
These are backend events that include the creation, read, update, deletion of records and other events that might be triggered from layers that aren't necessarily only available in the interface.
**Event funnels**
- How many sessions performed action A, B, then C
- What is our conversion rate from step A to B?
**PostgreSQL data**
These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql)
**Logs**
These are raw logs such as the [Production logs](../../administration/logs.md#production_jsonlog), [API logs](../../administration/logs.md#api_jsonlog), or [Sidekiq logs](../../administration/logs.md#sidekiqlog). See the [overview of Logging Infrastructure](https://gitlab.com/gitlab-com/runbooks/tree/master/logging/doc#logging-infrastructure-overview) for more details.
**External services**
These are external services a GitLab instance interacts with such as an [external storage provider](../../administration/static_objects_external_storage.md) or an [external container registry](../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). These services must be able to send data back into a GitLab instance for data to be tracked.
## Telemetry systems overview
The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed nstances.
For Telemetry purposes, GitLab Inc has three major components:
1.[Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/data-infrastructure/): This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors [GitLab.com's Snowplow Collector](https://about.gitlab.com/handbook/engineering/infrastructure/library/snowplow/) and GitLab's Versions Application.
1. GitLab.com: This is the production GitLab application which is made up of a Client and Server. On the Client or browser side, a Snowplow JS Tracker (Frontend) is used to track client-side events. On the Server or application side, a Snowplow Ruby Tracker (Backend) is used to track server-side events. The server also contains Usage Ping which leverages a PostgreSQL database and a Redis in-memory data store to report on usage data. Lastly, the server also contains System Logs which are generated from running the GitLab application.
1.[Monitoring infrastructure](https://about.gitlab.com/handbook/engineering/monitoring/): This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana
### Self-managed
For Telemetry purposes, self-managed instances have two major components:
1. Data infrastructure: Having a data infrastructure setup is optional on self-managed instances. If you'd like to collect Snowplow tracking events for your self-managed instance, you can setup your own self-managed Snowplow collector and configure your Snowplow events to point to your own collector.
1. GitLab: A self-managed GitLab instance contains all of the same components as GitLab.com mentioned above.
### Differences between GitLab Inc and Self-managed
As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Usage Ping, and PostgreSQL database imports all flow into GitLab Inc's data fnfrastructure. However, on self-managed, only Usage Ping flows into GitLab Inc's data infrastructure.
As shown by the green lines, on GitLab.com system logs flow into GitLab Inc's monitoring infrastructure. On self-managed, there are no logs sent to GitLab Inc's monitoring infrastructure.
The differences between GitLab.com and self-managed are summarized below:
Note (1): Snowplow JS and Snowplow Ruby are available on self-managed, however, the Snowplow Collector endpoint is set to a self-managed Snowplow Collector which GitLab Inc does not have access to.
Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application.
From [Snowplow's documentation](https://github.com/snowplow/snowplow), Snowplow consists of six loosely-coupled sub-systems:
-**Trackers** fire Snowplow events. Currently Snowplow has 12 trackers, covering web, mobile, desktop, server and IoT
-**Collectors** receive Snowplow events from trackers. Currently we have three different event collectors, sinking events either to Amazon S3, Apache Kafka or Amazon Kinesis
-**Enrich** cleans up the raw Snowplow events, enriches them and puts them into storage. Currently we have a Hadoop-based enrichment process, and a Kinesis- or Kafka-based process
-**Storage** is where the Snowplow events live. Currently we store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases
-**Data modeling** is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker
-**Analytics** are performed on the Snowplow events or on the aggregate tables.
> 
## Snowplow schema
We currently have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/issues/207930) including the following definitions:
- The instance level, which will enable tracking on both the frontend and backend layers.
- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser will also not be tracked from a user level.
We utilize Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to:
-**Admin Area > Settings > Integrations** in the UI.
-`admin/application_settings/integrations` in your browser.
The following configuration is required:
| Name | Value |
| ------------- | ------------------------- |
| Collector | `snowplow.trx.gitlab.net` |
| Site ID | `gitlab` |
| Cookie domain | `.gitlab.com` |
## Snowplow request flow
The following example shows a basic request/response flow between a Snowplow JS / Ruby Trackers on GitLab.com, [the GitLab.com Snowplow Collector](https://about.gitlab.com/handbook/engineering/infrastructure/library/snowplow/), GitLab's S3 Bucket, GitLab's Snowflake Data Warehouse, and Sisense.:
```mermaid
sequenceDiagram
participant Snowplow JS (Frontend)
participant Snowplow Ruby (Backend)
participant GitLab.com Snowplow Collector
participant S3 Bucket
participant Snowflake DW
participant Sisense Dashboards
Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event
Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event
loop Process using Kinesis Stream
GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events
Snowflake DW->>Snowflake DW: Transform data using dbt
Snowflake DW->>Sisense Dashboards: Data available for querying
```
## Implementing Snowplow JS (Frontend) tracking
GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilize tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy).
| `category` | string | document.body.dataset.page | Page or subsection of a page that events are being captured within. |
| `action` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. |
| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). |
### Tracking in HAML (or Vue Templates)
When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-event` attribute will automatically have event tracking bound on clicks.
Below is an example of `data-track-*` attributes assigned to a button:
Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows for them to be properly handled on rerendering and changes to the DOM, but it's important to know that because of the way these events are bound, click events shouldn't be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you'll need to implement your own listeners and follow the instructions in [Tracking in raw JavaScript](#tracking-in-raw-javascript).
Below is a list of supported `data-track-*` attributes:
| `data-track-event` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. |
| `data-track-label` | false | The `label` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). |
| `data-track-property` | false | The `property` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). |
| `data-track-value` | false | The `value` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). If omitted, this will be the element's `value` property or an empty string. For checkboxes, the default value will be the element's checked attribute or `false` when unchecked. |
| `data-track-context` | false | The `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). |
### Tracking within Vue components
There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin.
You can provide default options that will be passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default.
You can then use the mixin normally in your component with the `mixin`, Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These will override any defaults and allows the values to be dynamic from props, or based on state.
```javascript
exportdefault{
mixins:[trackingMixin],
// ...[component implementation]...
data(){
return{
expanded:false,
tracking:{
label:'left_sidebar'
}
};
},
}
```
The mixin provides a `track` method that can be called within the template, or from component methods. An example of the whole implementation might look like the following.
```javascript
exportdefault{
mixins:[Tracking.mixin({label:'right_sidebar'})],
data(){
return{
expanded:false,
};
},
methods:{
toggle(){
this.expanded=!this.expanded;
this.track('click_toggle',{value:this.expanded})
}
}
};
```
And if needed within the template, you can use the `track` method directly as well.
<a@click.prevent="track('click_action')">Track an event</a>
</div>
</div>
</template>
```
### Tracking in raw JavaScript
Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually.
GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events.
Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments:
| `category` | string | 'application' | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. |
| `action` | string | 'generic' | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. |
| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). These will be set as empty strings if you don't provide them. |
Tracking can be viewed as either tracking user behavior, or can be utilized for instrumentation to monitor and visual performance over time in an area or aspect of code.
We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development.
## Developing and testing Snowplow
There are several tools for developing and testing Snowplow Event
| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment |
1. Open Chrome DevTools to the Snowplow Analytics Debugger tab
1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html)
### Snowplow Inspector Chrome Extension
Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments.
1. Open the chrome extension by pressing the Snowplow Inspector icon beside the address bar
1. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window.
### Snowplow Micro
Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried.
Snowplow Micro is a docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up.
- GitLab sends a weekly payload containing usage data to GitLab Inc. The usage ping uses high-level data to help our product, support, and sales teams. It does not send any project names, usernames, or any other specific data. The information from the usage ping is not anonymous, it is linked to the hostname of the instance. Sending usage ping is optional, and any instance can disable analytics.
- The usage data is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product.
- Usage ping is important to GitLab as we use it to calculate our and Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features.
- Once usage ping is enabled, GitLab will gather data from the other instances and will be able to show usage statistics of your instance to your users.
### Why Should We Enable Usage Ping?
- The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we are able to make better product decisions.
- As a benefit of having the usage ping active, GitLab lets you analyze the users’ activities over time of your GitLab installation.
- As a benefit of having the usage ping active, GitLab provides you with The DevOps Score,which gives you an overview of your entire instance’s adoption of Concurrent DevOps from planning to monitoring.
- You will get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value)
- You will get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization?
- You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes.
### Limitations
- Usage Ping does not track frontend events things like page views, link clicks, or user sessions and only focuses on aggregated backend events.
- Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Usage Ping to track aggregated backend events on self-managed.
## Usage Ping payload
You can view the exact JSON payload sent to GitLab Inc. in the administration panel. To view the payload:
1. Navigate to the **Admin Area > Settings > Metrics and profiling**.
The usage ping is opt-out. If you want to deactivate this feature, go to the Settings page of your administration panel and uncheck the Usage Ping checkbox.
To disable the usage ping and prevent it from being configured in future through the administration panel, Omnibus installs can set the following in [`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options):
```ruby
gitlab_rails['usage_ping_enabled']=false
```
And source installs can set the following in `gitlab.yml`:
```yaml
production:&base
# ...
gitlab:
# ...
usage_ping_enabled:false
```
## Usage Ping Request Flow
The following example shows a basic request/response flow between a GitLab Instance, the Versions Application, the License Application, Salesforce, GitLab's S3 Bucket, GitLab's Snowflake Data Warehouse, and Sisense.:
Snowflake DW->>Snowflake DW: Transform data using dbt
Snowflake DW->>Sisense Dashboards: Data available for querying
Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index)
```
## How Usage Ping works
1. The Usage Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_usage_ping_worker.rb#L30) is set in Sidekiq to run weekly.
1. When the cron job runs, it calls [GitLab::UsageData.to_json](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L22).
1. GitLab::UsageData.to_json [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L22) to ~400+ other counter method calls.
1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L14) into a single JSON payload in GitLab::UsageData.to_json.
1. The JSON payload is then [posted to the Versions application](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L20).
## Implementing Usage Ping
Usage Ping consists of four types of counters which are all found in `usage_data.rb`:
-**Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation
-**Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation on given column
-**Alternative Counters:** Used for settings and configurations
-**Redis Counters:** Used for in-memory counts. This method is being deprecated due to data inaccuracies and will be replaced with a persistent method.
Note: Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Usage Ping.
### Why batch counting
For large tables, PostgreSQL can take a long time to count rows due to MVCC [(Multi-version Concurrency Control)](https://en.wikipedia.org/wiki/Multiversion_concurrency_control). Batch counting is a counting method where a single large query is broken into multiple smaller queries. For example, instead of a single query querying 1,000,000 records, with batch counting, you can execute 100 queries of 10,000 records each. Batch counting is useful for avoiding database timeouts as each batch query is significantly shorter than one single long running query.
For GitLab.com, there are extremely large tables with 15 second query timeouts, so, we use batch counting to avoid encountering timeouts. Here are the sizes of some GitLab.com tables:
| Table | Row counts in millions |
| ------ | ------ |
| merge_request_diff_commits | 2280 |
| ci_build_trace_sections | 1764 |
| merge_request_diff_files | 1082 |
| events | 514 |
There are two batch counting methods provided, `Ordinary Batch Counters` and `Distinct Batch Counters`. Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, a specialized index may need to be added on the columns involved in a counter.
Note that Redis counters are in the [process of being deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/216330) and you should instead try to use Snowplow events instead. We're in the process of building [self-managed event tracking](https://gitlab.com/gitlab-org/telemetry/-/issues/373) and once this is available, we will convert all Redis counters into Snowplow events.
### Alternative Counters
Handles `StandardError` and fallbacks into -1 this way not all measures fail if we encounter one exception.
Paste the SQL query into `#database-lab` to see how the query performs at scale.
- #database-lab is a Slack channel which uses a production-sized environment to test your queries
- GitLab.com’s production database has a 15 second timeout.
- For each query we require an execution time of under 1 second due do cold caches which can 10x this time.
- Add a specialized index on columns involved to reduce your the execution time.
In order to have an understanding of the queries execution we add in the MR description the following information
For counters that have a `time_period` test and add information for both cases.
- with `time_period = {}` for all time period
- and `time_period = { created_at: 28.days.ago..Time.current }` for last 28 days period
Execution plan and query time before and after optimization
Using database-lab and [explain.depesz.com](https://explain.depesz.com/) see more details in [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries)
On GitLab.com, we have DangerBot setup to monitor Telemetry related files and DangerBot will recommend a Telemetry review. Simply `@gitlab-org/growth/telemetry/engineers` in your MR for a review.
At GitLab, we encourage event tracking so we can iterate on and improve the project and user experience.
We do this by running experiments, and collecting analytics for features and feature variations. This is:
- So we generally know engagement.
- A way to approach A/B testing.
As developers, we should attempt to add tracking and instrumentation where possible. This enables the Product team to better understand:
- User engagement.
- Usage patterns.
- Other metrics that can potentially be improved on.
To maintain consistency and not adversely effect performance, we have some basic tracking functionality exposed at both the frontend and backend layers that can be utilized while building new features or updating existing features.
We also encourage users to enable tracking, and we embrace full transparency with our tracking approach so it can be easily understood and trusted. By enabling tracking, users can:
- Contribute back to the wider community.
- Help GitLab improve on the product.
## Implementing tracking
Event tracking can be implemented on either the frontend or the backend layers, and each can be approached slightly differently since they have slightly different concerns.
In GitLab, many actions can be initiated via the web interface, but they can also be initiated via an API client (an iOS applications is a good example of this), or via `git` directly. Crucially, this means that tracking should be considered holistically for the feature that's being instrumented.
The data team should be involved when defining analytics and can be consulted when coming up with ways of presenting data that's being tracked. This allows our event data to be considered carefully and presented in ways that may reveal details about user engagement that may not be fully understood or interactions where we can make improvements. You can [contact the data team](https://about.gitlab.com/handbook/business-ops/data-team/#contact-us) and consult with them when defining tracking strategies.
### Frontend
Generally speaking, the frontend can track user actions and events, like:
- Clicking links or buttons.
- Submitting forms.
- Other typically interface-driven actions.
### Backend
From the backend, the events that are tracked will likely consist of things like the creation or deletion of records and other events that might be triggered from layers that aren't necessarily only available in the interface.
Also, see [Instrumenting Ruby code](../development/instrumentation.md) if you are instrumenting application performance metrics for Ruby code.
## Enabling tracking
Tracking can be enabled at:
- The instance level, which will enable tracking on both the frontend and backend layers.
- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser will also not be tracked from a user level.
We utilize Snowplow for the majority of our tracking strategy, and it can be enabled by navigating to:
-**Admin Area > Settings > Integrations** in the UI.
-`admin/application_settings/integrations` in your browser.
The following configuration is required:
| Name | Value |
| ------------- | ------------------------- |
| Collector | `snowplow.trx.gitlab.net` |
| Site ID | `gitlab` |
| Cookie domain | `.gitlab.com` |
Once enabled, tracking events can be inspected locally by either:
- Looking at the network panel of the browser's development tools
- Using the [Snowplow Chrome Extension](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm).
This document was moved to [another location](../development/telemetry/index.md).
GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilize tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy).
| `category` | string | document.body.dataset.page | Page or subsection of a page that events are being captured within. |
| `action` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. |
| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). |
### Tracking in HAML (or Vue Templates)
When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-event` attribute will automatically have event tracking bound on clicks.
Below is an example of `data-track-*` attributes assigned to a button:
Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows for them to be properly handled on rerendering and changes to the DOM, but it's important to know that because of the way these events are bound, click events shouldn't be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you'll need to implement your own listeners and follow the instructions in [Tracking in raw JavaScript](#tracking-in-raw-javascript).
Below is a list of supported `data-track-*` attributes:
| `data-track-event` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. |
| `data-track-label` | false | The `label` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). |
| `data-track-property` | false | The `property` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). |
| `data-track-value` | false | The `value` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). If omitted, this will be the elements `value` property or an empty string. For checkboxes, the default value will be the element's checked attribute or `false` when unchecked. |
| `data-track-context` | false | The `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). |
### Tracking within Vue components
There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin.
You can provide default options that will be passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default.
You can then use the mixin normally in your component with the `mixin`, Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These will override any defaults and allows the values to be dynamic from props, or based on state.
```javascript
exportdefault{
mixins:[trackingMixin],
// ...[component implementation]...
data(){
return{
expanded:false,
tracking:{
label:'left_sidebar'
}
};
},
}
```
The mixin provides a `track` method that can be called within the template, or from component methods. An example of the whole implementation might look like the following.
```javascript
exportdefault{
mixins:[Tracking.mixin({label:'right_sidebar'})],
data(){
return{
expanded:false,
};
},
methods:{
toggle(){
this.expanded=!this.expanded;
this.track('click_toggle',{value:this.expanded})
}
}
};
```
And if needed within the template, you can use the `track` method directly as well.
<a@click.prevent="track('click_action')">Track an event</a>
</div>
</div>
</template>
```
### Tracking in raw JavaScript
Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually.
GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events.
### Tracking in Ruby
Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments:
| `category` | string | 'application' | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. |
| `action` | string | 'generic' | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. |
| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/product/feature-instrumentation/#taxonomy). These will be set as empty strings if you don't provide them. |
Tracking can be viewed as either tracking user behavior, or can be utilized for instrumentation to monitor and visual performance over time in an area or aspect of code.
We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development.
This document was moved to [another location](../development/telemetry/index.md).
@@ -35,7 +35,7 @@ Access the default page for admin area settings by navigating to
| [PlantUML](../../../administration/integration/plantuml.md#gitlab) | Allow rendering of PlantUML diagrams in Asciidoc documents. |
| [Slack application](../../../user/project/integrations/gitlab_slack_application.md#configuration)**(FREE ONLY)** | Slack integration allows you to interact with GitLab via slash commands in a chat window. This option is only available on GitLab.com, though it may be [available for self-managed instances in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/28164). |
| [Third party offers](third_party_offers.md) | Control the display of third party offers. |
| [Snowplow](../../../telemetry/index.md#enabling-tracking) | Configure the Snowplow integration. |
| [Snowplow](../../../development/telemetry/snowplow.md) | Configure the Snowplow integration. |
| [Google GKE](../../project/clusters/add_gke_clusters.md) | Google GKE integration allows you to provision GKE clusters from GitLab. |
| [Amazon EKS](../../project/clusters/add_eks_clusters.md) | Amazon EKS integration allows you to provision EKS clusters from GitLab. |
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/557) in GitLab Enterprise Edition 8.10.
> - More statistics [were added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/735) in GitLab Enterprise Edition 8.12.
> - [Moved to GitLab Core](https://gitlab.com/gitlab-org/gitlab-foss/issues/23361) in 9.1.
> - More statistics [were added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/6602) in GitLab Ultimate 11.2.
GitLab sends a weekly payload containing usage data to GitLab Inc. The usage
ping uses high-level data to help our product, support, and sales teams. It does
not send any project names, usernames, or any other specific data. The
information from the usage ping is not anonymous, it is linked to the hostname
of the instance.
You can view the exact JSON payload in the administration panel. To view the payload:
1. Navigate to the **Admin Area > Settings > Metrics and profiling**.
1. Expand the **Usage statistics** section.
1. Click the **Preview payload** button.
You can see how [the usage ping data maps to different stages of the product](https://gitlab.com/gitlab-data/analytics/blob/master/transform/snowflake-dbt/data/version_usage_stats_to_stage_mappings.csv).
Usage ping is important to GitLab as we use it to calculate our [Action Monthly Active Users (AMAU)](https://about.gitlab.com/handbook/product/metrics/#action-monthly-active-users-amau) which helps us measure the success of our features.
### Request flow example
The following example shows a basic request/response flow between the self-managed GitLab instance, GitLab Version Application,
GitLab License Application and Salesforce:
```mermaid
sequenceDiagram
participant GitLab instance
participant Version Application
participant License Application
participant Salesforce
GitLab instance->>Version Application: Usage Ping data
loop Process Usage Data
Version Application->>Version Application: Parse Usage Data
Version Application->>Version Application: Record Usage Data
Version Application->>Version Application: Update license ping time
end
Version Application-xLicense Application: Request Zuora subscription id
License Application-xVersion Application: Zuora subscription id
Version Application-xSalesforce: Request Zuora account id by Zuora subscription id
Salesforce-xVersion Application: Zuora account id
Version Application-xSalesforce: Usage data for the Zuora account
Version Application->>GitLab instance: Conversational Development Index
```
### Deactivate the usage ping
The usage ping is opt-out. If you want to deactivate this feature, go to
the Settings page of your administration panel and uncheck the Usage ping
checkbox.
To disable the usage ping and prevent it from being configured in future through
the administration panel, Omnibus installs can set the following in
