@@ -23,11 +23,11 @@ Integration with Prometheus requires the following:
## Getting started with Prometheus monitoring
Depending on your deployment and where you have located your Prometheus server, there are a few options to get started with Prometheus monitoring.
Depending on your deployment and where you have located your GitLab server, there are a few options to get started with Prometheus monitoring.
* If both GitLab and your applications are installed in the same Kubernetes cluster, you can leveraged the [bundled Prometheus server within GitLab](#configuring-omnibus-gitlab-prometheus-to-monitor-kubernetes).
* If your applications are deployed on Kubernetes, but GitLab is not in the same cluster, then you can [configure a Prometheus server in your Kubernetes cluster](#configuring-your-own-prometheus-server-within-kubernetes).
* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-proemtheus-outside-of-kubernetes).
* If your applications are not running in Kubernetes, [get started with Prometheus](#getting-started-with-prometheus-outside-of-kubernetes).
### Getting started with Prometheus outside of Kubernetes
Once deployed, you should see the Prometheus service, deployment, and
pod start within the `prometheus` namespace. The server will begin to collect
metrics from each Kubernetes Node in the cluster, based on the configuration
provided in the template.
provided in the template. It will also attempt to collect metrics from any Kubernetes Pods that have been [annotated for Prometheus](https://prometheus.io/docs/operating/configuration/#pod).
Since GitLab is not running within Kubernetes, the template provides external
network access via a `NodePort` running on `30090`. This method allows access
...
...
@@ -139,8 +139,9 @@ environment which has had a successful deployment.
> [Introduced][ce-10408] in GitLab 9.2.
> GitLab 9.3 added the [numeric comparison](https://gitlab.com/gitlab-org/gitlab-ce/issues/27439) of the 30 minute averages.
Developers can view theperformance impact of their changes within the merge
Developers can view theperformance impact of their changes within the merge
request workflow. When a source branch has been deployed to an environment, a sparkline and numeric comparison of the average memory consumption will appear. On the sparkline, a dot
indicates when the current changes were deployed, with up to 30 minutes of
performance data displayed before and after. The comparison shows the difference between the 30 minute average before and after the deployment. This information is updated after
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0
GitLab has support for automatically detecting and monitoring Kubernetes metrics. Kubernetes exposes Node level metrics out of the box via the built-in [Prometheus metrics support in cAdvisor](https://github.com/google/cadvisor).
GitLab has support for automatically detecting and monitoring Kubernetes metrics. Kubernetes exposes Node level metrics out of the box via the built-in [Prometheus metrics support in cAdvisor](https://github.com/google/cadvisor). No additional services or exporters are needed.
## Metrics supported
...
...
@@ -12,8 +12,15 @@ GitLab has support for automatically detecting and monitoring Kubernetes metrics
## Configuring Prometheus to monitor for Kubernetes node metrics
Prometheus has internal support for discovering and monitoring of Kubernetes node metrics.
In order for Prometheus to collect Kubernetes metrics, you first must have a
Prometheus server up and running. You have two options here:
If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes).
- If you have an Omnibus based GitLab installation within your Kubernetes cluster, you can leverage the bundled Prometheus server to [monitor Kubernetes](../../../../administration/monitoring/prometheus/index.md#configuring-prometheus-to-monitor-kubernetes).
- To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) or [our guide](../../../../administration/monitoring/prometheus/index.md#configuring-your-own-prometheus-server-within-kubernetes).
To configure your own Prometheus server
## Specifying the Environment label
In order to isolate and only display relevant metrics for a given environment
however, GitLab needs a method to detect which labels are associated. To do this, GitLab will [look for an `environment` label](metrics.md#identifying-environments).
If you are using [GitLab Auto-Deploy][autodeploy] and one of the two [provided Kubernetes monitoring solutions](../prometheus.md#getting-started-with-prometheus-monitoring), the `environment` label will be automatically added.
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0
GitLab offers
GitLab offers automatic detection of select [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/). Currently supported exporters are:
*[Kubernetes](kubernetes.md)
*[NGINX](nginx.md)
*[Amazon Cloud Watch](cloudwatch.md)
## Metrics and Labels
We have tried to surface the most important metrics for each exporter, and will be continuing to add support for additional exporters in future releases. If you would like to add support for other official exporters, [contributions](#adding-to-the-library) are welcome.
GitLab retrieves performance data from two metrics, `container_cpu_usage_seconds_total`
and `container_memory_usage_bytes`. These metrics are collected from the
Kubernetes pods via Prometheus, and report CPU and Memory utilization of each
container or Pod running in the cluster.
## Identifying Environments
In order to isolate and only display relevant metrics for a given environment
however, GitLab needs a method to detect which pods are associated. To do that,
GitLab will specifically request metrics that have an `environment` tag that
GitLab retrieves performance data from the configured Prometheus server, and attempts to identifying the presence of known metrics. Once identified, GitLab then needs to be able to map the data to a particular environment.
In order to isolate and only display relevant metrics for a given environment, GitLab needs a method to detect which labels are associated. To do that,
GitLab will look for the required metrics which have a label that
matches the [$CI_ENVIRONMENT_SLUG][ci-environment-slug].
If you are using [GitLab Auto-Deploy][autodeploy] and one of the methods of
configuring Prometheus above, the `environment` will be automatically added.
For example if you are deploying to an environment named `production`, there must be a label for the metric with the value of `production`.
## Configuring Prometheus to collect automatically collected metrics within Kubernetes
## Adding to the library
In order for Prometheus to collect Kubernetes metrics, you first must have a
Prometheus server up and running. You have two options here:
We strive to support the 2-4 most important metrics for each common system service that supports Prometheus. If you are looking for support for a particular exporter which has not yet been added to the library, additions can be made [to the `additional_metrics.yml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/prometheus/additional_metrics.yml) file.
- If you installed Omnibus GitLab inside of Kubernetes, you can simply use the
[bundled version of Prometheus][promgldocs]. In that case, follow the info in the
- If you are using GitLab.com or installed GitLab outside of Kubernetes, you
will likely need to run a Prometheus server within the Kubernetes cluster.
Once installed, the easiest way to monitor Kubernetes is to simply use
Prometheus' support for [Kubernetes Service Discovery][prometheus-k8s-sd].
In that case, follow the instructions on
[configuring your own Prometheus server within Kubernetes](../prometheus.md#configuring-your-own-prometheus-server-within-kubernetes).
> Note: The library is only for monitoring public, common, system services which all customers can benefit from. Support for monitoring [customer proprietary metrics](https://gitlab.com/gitlab-org/gitlab-ee/issues/2273) will be added in an subsequent release.
