Commit d907812c authored by Alex Ives's avatar Alex Ives Committed by Mike Kozono

Add metrics for terraform state store

- Refactor geo_node_status so that it gathers metrics
  automatically
- Add specs that iterate over replicable classes with
  generated metrics
- Add `checksummed` traits to tf states
- Update framework docs to reflect changes
- Update sidekiq metrics docs to remove `_count` from
  geo fields in prometheus
- Generate the metrics in the geo_node_statuses factory

Relates to https://gitlab.com/gitlab-org/gitlab/issues/228918
parent 1e32cb57
...@@ -157,25 +157,34 @@ configuration option in `gitlab.yml`. These metrics are served from the ...@@ -157,25 +157,34 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `geo_lfs_objects_synced_missing_on_primary` | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | `url` | | `geo_lfs_objects_synced_missing_on_primary` | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | `url` |
| `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | `url` | | `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | `url` |
| `geo_attachments_synced_missing_on_primary` | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | `url` | | `geo_attachments_synced_missing_on_primary` | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | `url` |
| `geo_repositories_checksummed_count` | Gauge | 10.7 | Number of repositories checksummed on primary | `url` | | `geo_repositories_checksummed` | Gauge | 10.7 | Number of repositories checksummed on primary | `url` |
| `geo_repositories_checksum_failed_count` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` | | `geo_repositories_checksum_failed` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` |
| `geo_wikis_checksummed_count` | Gauge | 10.7 | Number of wikis checksummed on primary | `url` | | `geo_wikis_checksummed` | Gauge | 10.7 | Number of wikis checksummed on primary | `url` |
| `geo_wikis_checksum_failed_count` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | `url` | | `geo_wikis_checksum_failed` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | `url` |
| `geo_repositories_verified_count` | Gauge | 10.7 | Number of repositories verified on secondary | `url` | | `geo_repositories_verified` | Gauge | 10.7 | Number of repositories verified on secondary | `url` |
| `geo_repositories_verification_failed_count` | Gauge | 10.7 | Number of repositories failed to verify on secondary | `url` | | `geo_repositories_verification_failed` | Gauge | 10.7 | Number of repositories failed to verify on secondary | `url` |
| `geo_repositories_checksum_mismatch_count` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | `url` | | `geo_repositories_checksum_mismatch` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | `url` |
| `geo_wikis_verified_count` | Gauge | 10.7 | Number of wikis verified on secondary | `url` | | `geo_wikis_verified` | Gauge | 10.7 | Number of wikis verified on secondary | `url` |
| `geo_wikis_verification_failed_count` | Gauge | 10.7 | Number of wikis failed to verify on secondary | `url` | | `geo_wikis_verification_failed` | Gauge | 10.7 | Number of wikis failed to verify on secondary | `url` |
| `geo_wikis_checksum_mismatch_count` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | `url` | | `geo_wikis_checksum_mismatch` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | `url` |
| `geo_repositories_checked_count` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | `url` | | `geo_repositories_checked` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | `url` |
| `geo_repositories_checked_failed_count` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | `url` | | `geo_repositories_checked_failed` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | `url` |
| `geo_repositories_retrying_verification_count` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | `url` | | `geo_repositories_retrying_verification` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | `url` |
| `geo_wikis_retrying_verification_count` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | `url` | | `geo_wikis_retrying_verification` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | `url` |
| `geo_package_files` | Gauge | 13.0 | Number of package files on primary | `url` |
| `geo_package_files_checksummed` | Gauge | 13.0 | Number of package files checksummed on primary | `url` |
| `geo_package_files_checksum_failed` | Gauge | 13.0 | Number of package files failed to calculate the checksum on primary | `url` |
| `geo_package_files_synced` | Gauge | 13.3 | Number of syncable package files synced on secondary | `url` |
| `geo_package_files_failed` | Gauge | 13.3 | Number of syncable package files failed to sync on secondary | `url` |
| `geo_package_files_registry` | Gauge | 13.3 | Number of package files in the registry | `url` |
| `geo_terraform_states` | Gauge | 13.3 | Number of terraform states on primary | `url` |
| `geo_terraform_states_checksummed` | Gauge | 13.3 | Number of terraform states checksummed on primary | `url` |
| `geo_terraform_states_checksum_failed` | Gauge | 13.3 | Number of terraform states failed to calculate the checksum on primary | `url` |
| `geo_terraform_states_synced` | Gauge | 13.3 | Number of syncable terraform states synced on secondary | `url` |
| `geo_terraform_states_failed` | Gauge | 13.3 | Number of syncable terraform states failed to sync on secondary | `url` |
| `geo_terraform_states_registry` | Gauge | 13.3 | Number of terraform states in the registry | `url` |
| `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | | | `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | |
| `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | | | `global_search_awaiting_indexing_queue_size` | Gauge | 13.2 | Number of database updates waiting to be synchronized to Elasticsearch while indexing is paused | |
| `package_files_count` | Gauge | 13.0 | Number of package files on primary | `url` |
| `package_files_checksummed_count` | Gauge | 13.0 | Number of package files checksummed on primary | `url` |
| `package_files_checksum_failed_count` | Gauge | 13.0 | Number of package files failed to calculate the checksum on primary
## Database load balancing metrics **(PREMIUM ONLY)** ## Database load balancing metrics **(PREMIUM ONLY)**
......
...@@ -447,7 +447,13 @@ Example response: ...@@ -447,7 +447,13 @@ Example response:
"package_files_checksum_failed_count": 0, "package_files_checksum_failed_count": 0,
"package_files_registry_count": 10, "package_files_registry_count": 10,
"package_files_synced_count": 6, "package_files_synced_count": 6,
"package_files_failed_count": 3 "package_files_failed_count": 3,
"terraform_states_count": 10,
"terraform_states_checksummed_count": 10,
"terraform_states_checksum_failed_count": 0,
"terraform_states_registry_count": 10,
"terraform_states_synced_count": 6,
"terraform_states_failed_count": 3
} }
] ]
``` ```
......
...@@ -520,44 +520,37 @@ Widgets should now be verified by Geo! ...@@ -520,44 +520,37 @@ Widgets should now be verified by Geo!
Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in
`GeoNodeStatus` for display in the UI, and sent to Prometheus. `GeoNodeStatus` for display in the UI, and sent to Prometheus.
1. Add fields `widget_count`, `widget_checksummed_count`, 1. Add fields `widgets_count`, `widgets_checksummed_count`,
`widget_checksum_failed_count`, `widget_synced_count`, `widgets_checksum_failed_count`, `widgets_synced_count`,
`widget_failed_count`, and `widget_registry_count` to `widgets_failed_count`, and `widgets_registry_count` to
`GeoNodeStatus#RESOURCE_STATUS_FIELDS` array in `GET /geo_nodes/status` example response in
`ee/app/models/geo_node_status.rb`. `doc/api/geo_nodes.md`.
1. Add the same fields to `GeoNodeStatus#PROMETHEUS_METRICS` hash in
`ee/app/models/geo_node_status.rb`.
1. Add the same fields to `Sidekiq metrics` table in
`doc/administration/monitoring/prometheus/gitlab_metrics.md`.
1. Add the same fields to `GET /geo_nodes/status` example response in 1. Add the same fields to `GET /geo_nodes/status` example response in
`doc/api/geo_nodes.md`. `doc/api/geo_nodes.md`.
1. Add the same fields to `ee/spec/models/geo_node_status_spec.rb` and 1. Add fields `geo_widgets`, `geo_widgets_checksummed`,
`ee/spec/factories/geo_node_statuses.rb`. `geo_widgets_checksum_failed`, `geo_widgets_synced`,
1. Set `widget_count` in `GeoNodeStatus#load_data_from_current_node`: `geo_widgets_failed`, and `geo_widgets_registry` to
`Sidekiq metrics` table in
`doc/administration/monitoring/prometheus/gitlab_metrics.md`.
1. Add the following to the parameterized table in
`ee/spec/models/geo_node_status_spec.rb`:
```ruby ```ruby
self.widget_count = Geo::WidgetReplicator.primary_total_count Geo::WidgetReplicator | :widget | :geo_widget_registry
``` ```
1. Add `GeoNodeStatus#load_widgets_data` to set `widget_synced_count`, 1. Add the following to `spec/factories/widgets.rb`:
`widget_failed_count`, and `widget_registry_count`:
```ruby ```ruby
def load_widget_data trait(:checksummed) do
self.widget_synced_count = Geo::WidgetReplicator.synced_count with_file
self.widget_failed_count = Geo::WidgetReplicator.failed_count verification_checksum { 'abc' }
self.widget_registry_count = Geo::WidgetReplicator.registry_count
end end
```
1. Call `GeoNodeStatus#load_widgets_data` in trait(:checksum_failure) do
`GeoNodeStatus#load_secondary_data`. with_file
verification_failure { 'Could not calculate the checksum' }
1. Set `widget_checksummed_count` and `widget_checksum_failed_count` in end
`GeoNodeStatus#load_verification_data`:
```ruby
self.widget_checksummed_count = Geo::WidgetReplicator.checksummed_count self.widget_checksum_failed_count = Geo::WidgetReplicator.checksum_failed_count
``` ```
Widget replication and verification metrics should now be available in the API, Widget replication and verification metrics should now be available in the API,
......
...@@ -31,7 +31,25 @@ class GeoNodeStatus < ApplicationRecord ...@@ -31,7 +31,25 @@ class GeoNodeStatus < ApplicationRecord
alias_attribute :last_event_timestamp, :last_event_date_timestamp alias_attribute :last_event_timestamp, :last_event_date_timestamp
alias_attribute :cursor_last_event_timestamp, :cursor_last_event_date_timestamp alias_attribute :cursor_last_event_timestamp, :cursor_last_event_date_timestamp
RESOURCE_STATUS_FIELDS = %w( def self.status_fields_for(replicable_class)
{
"#{replicable_class.replicable_name_plural}_count".to_sym => "Number of #{replicable_class.replicable_title_plural} on the primary",
"#{replicable_class.replicable_name_plural}_checksummed_count".to_sym => "Number of #{replicable_class.replicable_title_plural} checksummed on the primary",
"#{replicable_class.replicable_name_plural}_checksum_failed_count".to_sym => "Number of #{replicable_class.replicable_title_plural} failed to checksum on primary",
"#{replicable_class.replicable_name_plural}_synced_count".to_sym => "Number of #{replicable_class.replicable_title_plural} in the registry",
"#{replicable_class.replicable_name_plural}_failed_count".to_sym => "Number of #{replicable_class.replicable_title_plural} synced on secondary",
"#{replicable_class.replicable_name_plural}_registry_count".to_sym => "Number of #{replicable_class.replicable_title_plural} failed to sync on secondary"
}
end
# Why are disabled classes included? See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38959/diffs#note_402656534
def self.replicator_class_status_fields
Gitlab::Geo::REPLICATOR_CLASSES.map do |replicable_class|
status_fields_for(replicable_class).keys
end.flatten.map(&:to_s)
end
RESOURCE_STATUS_FIELDS = (%w(
repository_verification_enabled repository_verification_enabled
repositories_replication_enabled repositories_replication_enabled
repositories_synced_count repositories_synced_count
...@@ -75,13 +93,14 @@ class GeoNodeStatus < ApplicationRecord ...@@ -75,13 +93,14 @@ class GeoNodeStatus < ApplicationRecord
design_repositories_count design_repositories_count
design_repositories_synced_count design_repositories_synced_count
design_repositories_failed_count design_repositories_failed_count
package_files_count ) + replicator_class_status_fields).freeze
package_files_checksummed_count
package_files_checksum_failed_count # Why are disabled classes included? See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38959/diffs#note_402656534
package_files_synced_count def self.replicator_class_prometheus_metrics
package_files_failed_count Gitlab::Geo::REPLICATOR_CLASSES.map do |replicable_class|
package_files_registry_count status_fields_for(replicable_class)
).freeze end.reduce({}, :merge)
end
# Be sure to keep this consistent with Prometheus naming conventions # Be sure to keep this consistent with Prometheus naming conventions
PROMETHEUS_METRICS = { PROMETHEUS_METRICS = {
...@@ -153,14 +172,8 @@ class GeoNodeStatus < ApplicationRecord ...@@ -153,14 +172,8 @@ class GeoNodeStatus < ApplicationRecord
design_repositories_count: 'Total number of syncable design repositories available on primary', design_repositories_count: 'Total number of syncable design repositories available on primary',
design_repositories_synced_count: 'Number of syncable design repositories synced on secondary', design_repositories_synced_count: 'Number of syncable design repositories synced on secondary',
design_repositories_failed_count: 'Number of syncable design repositories failed to sync on secondary', design_repositories_failed_count: 'Number of syncable design repositories failed to sync on secondary',
design_repositories_registry_count: 'Number of design repositories in the registry', design_repositories_registry_count: 'Number of design repositories in the registry'
package_files_count: 'Number of package files on primary', }.merge(replicator_class_prometheus_metrics).freeze
package_files_checksummed_count: 'Number of package files checksummed on primary',
package_files_checksum_failed_count: 'Number of package files failed to checksum on primary',
package_files_registry_count: 'Number of package files in the registry',
package_files_synced_count: 'Number of package files synced on secondary',
package_files_failed_count: 'Number of package files failed to sync on secondary'
}.freeze
EXPIRATION_IN_MINUTES = 10 EXPIRATION_IN_MINUTES = 10
HEALTHY_STATUS = 'Healthy'.freeze HEALTHY_STATUS = 'Healthy'.freeze
...@@ -359,6 +372,14 @@ class GeoNodeStatus < ApplicationRecord ...@@ -359,6 +372,14 @@ class GeoNodeStatus < ApplicationRecord
end end
end end
def self.add_attr_in_percentage_for_replicable_classes
Gitlab::Geo::REPLICATOR_CLASSES.each do |replicator|
replicable = replicator.replicable_name_plural
attr_in_percentage "#{replicable}_checksummed", "#{replicable}_checksummed_count", "#{replicable}_count"
attr_in_percentage "#{replicable}_synced", "#{replicable}_synced_count", "#{replicable}_registry_count"
end
end
attr_in_percentage :repositories_synced, :repositories_synced_count, :repositories_count attr_in_percentage :repositories_synced, :repositories_synced_count, :repositories_count
attr_in_percentage :repositories_checksummed, :repositories_checksummed_count, :repositories_count attr_in_percentage :repositories_checksummed, :repositories_checksummed_count, :repositories_count
attr_in_percentage :repositories_verified, :repositories_verified_count, :repositories_count attr_in_percentage :repositories_verified, :repositories_verified_count, :repositories_count
...@@ -372,8 +393,8 @@ class GeoNodeStatus < ApplicationRecord ...@@ -372,8 +393,8 @@ class GeoNodeStatus < ApplicationRecord
attr_in_percentage :replication_slots_used, :replication_slots_used_count, :replication_slots_count attr_in_percentage :replication_slots_used, :replication_slots_used_count, :replication_slots_count
attr_in_percentage :container_repositories_synced, :container_repositories_synced_count, :container_repositories_count attr_in_percentage :container_repositories_synced, :container_repositories_synced_count, :container_repositories_count
attr_in_percentage :design_repositories_synced, :design_repositories_synced_count, :design_repositories_count attr_in_percentage :design_repositories_synced, :design_repositories_synced_count, :design_repositories_count
attr_in_percentage :package_files_checksummed, :package_files_checksummed_count, :package_files_count
attr_in_percentage :package_files_synced, :package_files_synced_count, :package_files_registry_count add_attr_in_percentage_for_replicable_classes
def synced_in_percentage_for(replicator_class) def synced_in_percentage_for(replicator_class)
public_send("#{replicator_class.replicable_name_plural}_synced_in_percentage") # rubocop:disable GitlabSecurity/PublicSend public_send("#{replicator_class.replicable_name_plural}_synced_in_percentage") # rubocop:disable GitlabSecurity/PublicSend
...@@ -422,10 +443,13 @@ class GeoNodeStatus < ApplicationRecord ...@@ -422,10 +443,13 @@ class GeoNodeStatus < ApplicationRecord
return unless Gitlab::Geo.primary? return unless Gitlab::Geo.primary?
self.projects_count = geo_node.projects.count self.projects_count = geo_node.projects.count
self.package_files_count = Geo::PackageFileReplicator.primary_total_count
self.replication_slots_count = geo_node.replication_slots_count self.replication_slots_count = geo_node.replication_slots_count
self.replication_slots_used_count = geo_node.replication_slots_used_count self.replication_slots_used_count = geo_node.replication_slots_used_count
self.replication_slots_max_retained_wal_bytes = geo_node.replication_slots_max_retained_wal_bytes self.replication_slots_max_retained_wal_bytes = geo_node.replication_slots_max_retained_wal_bytes
Gitlab::Geo::REPLICATOR_CLASSES.each do |replicator|
public_send("#{replicator.replicable_name_plural}_count=", replicator.primary_total_count) # rubocop:disable GitlabSecurity/PublicSend
end
end end
def load_secondary_data def load_secondary_data
...@@ -441,7 +465,7 @@ class GeoNodeStatus < ApplicationRecord ...@@ -441,7 +465,7 @@ class GeoNodeStatus < ApplicationRecord
load_attachments_data load_attachments_data
load_container_registry_data load_container_registry_data
load_designs_data load_designs_data
load_package_files_data load_ssf_replicable_data
end end
def load_repositories_data def load_repositories_data
...@@ -500,13 +524,13 @@ class GeoNodeStatus < ApplicationRecord ...@@ -500,13 +524,13 @@ class GeoNodeStatus < ApplicationRecord
self.design_repositories_registry_count = design_registry_finder.registry_count self.design_repositories_registry_count = design_registry_finder.registry_count
end end
def load_package_files_data def load_ssf_replicable_data
# return unless package_files_replication_enabled # TODO: https://gitlab.com/gitlab-org/gitlab/-/issues/221069 Gitlab::Geo::REPLICATOR_CLASSES.each do |replicator|
public_send("#{replicator.replicable_name_plural}_count=", replicator.registry_count) # rubocop:disable GitlabSecurity/PublicSend
self.package_files_count = Geo::PackageFileReplicator.registry_count public_send("#{replicator.replicable_name_plural}_registry_count=", replicator.registry_count) # rubocop:disable GitlabSecurity/PublicSend
self.package_files_registry_count = Geo::PackageFileReplicator.registry_count public_send("#{replicator.replicable_name_plural}_synced_count=", replicator.synced_count) # rubocop:disable GitlabSecurity/PublicSend
self.package_files_synced_count = Geo::PackageFileReplicator.synced_count public_send("#{replicator.replicable_name_plural}_failed_count=", replicator.failed_count) # rubocop:disable GitlabSecurity/PublicSend
self.package_files_failed_count = Geo::PackageFileReplicator.failed_count end
end end
def load_repository_check_data def load_repository_check_data
...@@ -534,8 +558,11 @@ class GeoNodeStatus < ApplicationRecord ...@@ -534,8 +558,11 @@ class GeoNodeStatus < ApplicationRecord
self.repositories_checksum_failed_count = repository_verification_finder.count_verification_failed_repositories self.repositories_checksum_failed_count = repository_verification_finder.count_verification_failed_repositories
self.wikis_checksummed_count = repository_verification_finder.count_verified_wikis self.wikis_checksummed_count = repository_verification_finder.count_verified_wikis
self.wikis_checksum_failed_count = repository_verification_finder.count_verification_failed_wikis self.wikis_checksum_failed_count = repository_verification_finder.count_verification_failed_wikis
self.package_files_checksummed_count = Geo::PackageFileReplicator.checksummed_count
self.package_files_checksum_failed_count = Geo::PackageFileReplicator.checksum_failed_count Gitlab::Geo::REPLICATOR_CLASSES.each do |replicator|
public_send("#{replicator.replicable_name_plural}_checksummed_count=", replicator.checksummed_count) # rubocop:disable GitlabSecurity/PublicSend
public_send("#{replicator.replicable_name_plural}_checksum_failed_count=", replicator.checksum_failed_count) # rubocop:disable GitlabSecurity/PublicSend
end
end end
def load_secondary_verification_data def load_secondary_verification_data
......
---
title: Add metrics for terraform state replication
merge_request: 38959
author:
type: added
...@@ -16,6 +16,14 @@ module Gitlab ...@@ -16,6 +16,14 @@ module Gitlab
API_SCOPE = 'geo_api' API_SCOPE = 'geo_api'
# TODO: Avoid having to maintain a list. Discussions related to possible
# solutions can be found at
# https://gitlab.com/gitlab-org/gitlab/-/issues/227693
REPLICATOR_CLASSES = [
::Geo::PackageFileReplicator,
::Geo::TerraformStateReplicator
].freeze
def self.current_node def self.current_node
self.cache_value(:current_node, as: GeoNode) { GeoNode.current_node } self.cache_value(:current_node, as: GeoNode) { GeoNode.current_node }
end end
...@@ -158,16 +166,8 @@ module Gitlab ...@@ -158,16 +166,8 @@ module Gitlab
_(template) % { url: url } _(template) % { url: url }
end end
# TODO: Avoid having to maintain a list. Discussions related to possible
# solutions can be found at
# https://gitlab.com/gitlab-org/gitlab/-/issues/227693
def self.replicator_classes def self.replicator_classes
classes = [ REPLICATOR_CLASSES.select(&:enabled?)
::Geo::PackageFileReplicator,
::Geo::TerraformStateReplicator
]
classes.select(&:enabled?)
end end
end end
end end
...@@ -37,12 +37,6 @@ FactoryBot.define do ...@@ -37,12 +37,6 @@ FactoryBot.define do
repositories_checksum_failed_count { 120 } repositories_checksum_failed_count { 120 }
wikis_checksummed_count { 585 } wikis_checksummed_count { 585 }
wikis_checksum_failed_count { 55 } wikis_checksum_failed_count { 55 }
package_files_count { 585 }
package_files_checksummed_count { 585 }
package_files_registry_count { 585 }
package_files_failed_count { 3 }
package_files_synced_count { 580 }
package_files_checksum_failed_count { 585 }
repositories_verified_count { 501 } repositories_verified_count { 501 }
repositories_verification_failed_count { 100 } repositories_verification_failed_count { 100 }
repositories_checksum_mismatch_count { 15 } repositories_checksum_mismatch_count { 15 }
...@@ -66,6 +60,10 @@ FactoryBot.define do ...@@ -66,6 +60,10 @@ FactoryBot.define do
lfs_objects_replication_enabled { true } lfs_objects_replication_enabled { true }
repositories_replication_enabled { true } repositories_replication_enabled { true }
repository_verification_enabled { true } repository_verification_enabled { true }
GeoNodeStatus.replicator_class_status_fields.each do |field|
send(field) { rand(10000) }
end
end end
trait :replicated_and_verified do trait :replicated_and_verified do
......
...@@ -49,6 +49,19 @@ ...@@ -49,6 +49,19 @@
"package_files_count", "package_files_count",
"package_files_checksum_failed_count", "package_files_checksum_failed_count",
"package_files_checksummed_in_percentage", "package_files_checksummed_in_percentage",
"package_files_checksummed_count",
"package_files_registry_count",
"package_files_failed_count",
"package_files_synced_count",
"package_files_synced_in_percentage",
"terraform_states_count",
"terraform_states_checksum_failed_count",
"terraform_states_checksummed_in_percentage",
"terraform_states_checksummed_count",
"terraform_states_registry_count",
"terraform_states_failed_count",
"terraform_states_synced_count",
"terraform_states_synced_in_percentage",
"repositories_verified_count", "repositories_verified_count",
"repositories_verification_failed_count", "repositories_verification_failed_count",
"repositories_verified_in_percentage", "repositories_verified_in_percentage",
...@@ -138,6 +151,14 @@ ...@@ -138,6 +151,14 @@
"package_files_synced_count": { "type": ["integer", "null"] }, "package_files_synced_count": { "type": ["integer", "null"] },
"package_files_synced_in_percentage": { "type": "string" }, "package_files_synced_in_percentage": { "type": "string" },
"package_files_checksummed_in_percentage": { "type": "string" }, "package_files_checksummed_in_percentage": { "type": "string" },
"terraform_states_count": { "type": ["integer", "null"] },
"terraform_states_checksummed_count": { "type": ["integer", "null"] },
"terraform_states_checksum_failed_count": { "type": ["integer", "null"] },
"terraform_states_registry_count": { "type": ["integer", "null"] },
"terraform_states_failed_count": { "type": ["integer", "null"] },
"terraform_states_synced_count": { "type": ["integer", "null"] },
"terraform_states_synced_in_percentage": { "type": "string" },
"terraform_states_checksummed_in_percentage": { "type": "string" },
"repositories_verified_count": { "type": ["integer", "null"] }, "repositories_verified_count": { "type": ["integer", "null"] },
"repositories_verification_failed_count": { "type": ["integer", "null"] }, "repositories_verification_failed_count": { "type": ["integer", "null"] },
"repositories_verified_in_percentage": { "type": "string" }, "repositories_verified_in_percentage": { "type": "string" },
......
...@@ -4,6 +4,7 @@ require 'spec_helper' ...@@ -4,6 +4,7 @@ require 'spec_helper'
RSpec.describe GeoNodeStatus, :geo do RSpec.describe GeoNodeStatus, :geo do
include ::EE::GeoHelpers include ::EE::GeoHelpers
using RSpec::Parameterized::TableSyntax
let!(:primary) { create(:geo_node, :primary) } let!(:primary) { create(:geo_node, :primary) }
let!(:secondary) { create(:geo_node) } let!(:secondary) { create(:geo_node) }
...@@ -1145,84 +1146,100 @@ RSpec.describe GeoNodeStatus, :geo do ...@@ -1145,84 +1146,100 @@ RSpec.describe GeoNodeStatus, :geo do
end end
end end
describe '#package_files_checksummed_count' do where(:replicator, :model_factory, :registry_factory) do
before do Geo::PackageFileReplicator | :package_file | :geo_package_file_registry
stub_current_geo_node(primary) Geo::TerraformStateReplicator | :terraform_state | :geo_terraform_state_registry
end
it 'returns the right number of checksummed package files' do
create(:package_file, :jar, :checksummed)
create(:package_file, :jar, :checksummed)
create(:package_file, :jar, :checksum_failure)
expect(subject.package_files_checksummed_count).to eq(2)
end
end end
describe '#package_files_checksum_failed_count' do with_them do
before do let(:replicable_name) { replicator.replicable_name_plural }
stub_current_geo_node(primary) let(:checksummed_count_method) { "#{replicable_name}_checksummed_count" }
end let(:checksum_failed_count_method) { "#{replicable_name}_checksum_failed_count" }
let(:checksummed_in_percentage_method) { "#{replicable_name}_checksummed_in_percentage" }
let(:registry_count_method) { "#{replicable_name}_registry_count" }
let(:failed_count_method) { "#{replicable_name}_failed_count" }
let(:synced_count_method) { "#{replicable_name}_synced_count" }
let(:synced_in_percentage_method) { "#{replicable_name}_synced_in_percentage" }
it 'returns the right number of failed package files' do describe '#<replicable_name>_checksummed_count' do
create(:package_file, :jar, :checksummed) before do
create(:package_file, :jar, :checksum_failure) stub_current_geo_node(primary)
create(:package_file, :jar, :checksum_failure) end
expect(subject.package_files_checksum_failed_count).to eq(2) it 'returns the right number of checksummed replicables' do
end create(model_factory, :checksummed)
end create(model_factory, :checksummed)
create(model_factory, :checksum_failure)
describe '#package_files_checksummed_in_percentage' do expect(subject.send(checksummed_count_method)).to eq(2)
before do end
stub_current_geo_node(primary)
end end
it 'returns 0 when no package files available' do describe '#<replicable_name>_checksum_failed_count' do
expect(subject.package_files_checksummed_in_percentage).to eq(0) before do
end stub_current_geo_node(primary)
end
it 'returns the right percentage' do it 'returns the right number of failed replicables' do
create(:package_file, :jar, :checksummed) create(model_factory, :checksummed)
create(:package_file, :jar, :checksummed) create(model_factory, :checksum_failure)
create(:package_file, :jar, :checksummed) create(model_factory, :checksum_failure)
create(:package_file, :jar, :checksum_failure)
expect(subject.package_files_checksummed_in_percentage).to be_within(0.0001).of(75) expect(subject.send(checksum_failed_count_method)).to eq(2)
end
end end
end
describe 'package files secondary counters' do describe '#<replicable_name>_checksummed_in_percentage' do
context 'when package registries available' do
before do before do
create(:geo_package_file_registry, :failed) stub_current_geo_node(primary)
create(:geo_package_file_registry, :failed)
create(:geo_package_file_registry, :synced)
end end
it 'returns the right number of repos in registry' do it 'returns 0 when no replicables available' do
expect(subject.package_files_registry_count).to eq(3) expect(subject.send(checksummed_in_percentage_method)).to eq(0)
end end
it 'returns the right number of failed and synced repos' do it 'returns the right percentage' do
expect(subject.package_files_failed_count).to eq(2) create(model_factory, :checksummed)
expect(subject.package_files_synced_count).to eq(1) create(model_factory, :checksummed)
end create(model_factory, :checksummed)
create(model_factory, :checksum_failure)
it 'returns the percent of synced package files' do expect(subject.send(checksummed_in_percentage_method)).to be_within(0.0001).of(75)
expect(subject.package_files_synced_in_percentage).to be_within(0.01).of(33.33)
end end
end end
context 'when no package registries available' do describe '#<replicable_name>_[registry|synced|failed]_count' do
it 'returns 0' do context 'when package registries available' do
expect(subject.package_files_registry_count).to eq(0) before do
expect(subject.package_files_failed_count).to eq(0) create(registry_factory, :failed)
expect(subject.package_files_synced_count).to eq(0) create(registry_factory, :failed)
create(registry_factory, :synced)
end
it 'returns the right number of repos in registry' do
expect(subject.send(registry_count_method)).to eq(3)
end
it 'returns the right number of failed and synced repos' do
expect(subject.send(failed_count_method)).to eq(2)
expect(subject.send(synced_count_method)).to eq(1)
end
it 'returns the percent of synced replicables' do
expect(subject.send(synced_in_percentage_method)).to be_within(0.01).of(33.33)
end
end end
it 'returns 0' do context 'when no package registries available' do
expect(subject.package_files_synced_in_percentage).to eq(0) it 'returns 0' do
expect(subject.send(registry_count_method)).to eq(0)
expect(subject.send(failed_count_method)).to eq(0)
expect(subject.send(synced_count_method)).to eq(0)
end
it 'returns 0' do
expect(subject.send(synced_in_percentage_method)).to eq(0)
end
end end
end end
end end
......
...@@ -15,5 +15,15 @@ FactoryBot.define do ...@@ -15,5 +15,15 @@ FactoryBot.define do
locked_at { Time.current } locked_at { Time.current }
locked_by_user { create(:user) } locked_by_user { create(:user) }
end end
trait(:checksummed) do
with_file
verification_checksum { 'abc' }
end
trait(:checksum_failure) do
with_file
verification_failure { 'Could not calculate the checksum' }
end
end end
end end
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment