Commit fbb76070 authored by Maxime Orefice's avatar Maxime Orefice Committed by Douglas Barbosa Alexandre

Add pipeline artifact to geo

This commit allows pipeline artifacts to be replicable with geo.
It has the necessary logic to replicate and verify the data.
parent 7c79b0da
......@@ -57,3 +57,5 @@ module Ci
end
end
end
Ci::PipelineArtifact.prepend_ee_mod
......@@ -42,3 +42,5 @@ module Ci
end
end
end
Ci::Artifactable.prepend_ee_mod
---
name: geo_pipeline_artifact_replication
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57741
rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/326228
milestone: '13.11'
type: development
group: group::geo
default_enabled: false
......@@ -24,6 +24,7 @@ ActiveSupport::Inflector.inflections do |inflect|
job_artifact_registry
lfs_object_registry
package_file_registry
pipeline_artifact_registry
project_auto_devops
project_registry
project_statistics
......
......@@ -432,6 +432,18 @@ Example response:
"group_wiki_repositories_verification_failed_count": null,
"group_wiki_repositories_synced_in_percentage": "0.00%",
"group_wiki_repositories_verified_in_percentage": "0.00%",
"pipeline_artifacts_count": 5,
"pipeline_artifacts_checksum_total_count": 5,
"pipeline_artifacts_checksummed_count": 5,
"pipeline_artifacts_checksum_failed_count": 0,
"pipeline_artifacts_synced_count": null,
"pipeline_artifacts_failed_count": null,
"pipeline_artifacts_registry_count": null,
"pipeline_artifacts_verification_total_count": null,
"pipeline_artifacts_verified_count": null,
"pipeline_artifacts_verification_failed_count": null,
"pipeline_artifacts_synced_in_percentage": "0.00%",
"pipeline_artifacts_verified_in_percentage": "0.00%",
},
{
"geo_node_id": 2,
......@@ -562,6 +574,18 @@ Example response:
"group_wiki_repositories_verification_failed_count": 0,
"group_wiki_repositories_synced_in_percentage": "100.00%",
"group_wiki_repositories_verified_in_percentage": "100.00%",
"pipeline_artifacts_count": 5,
"pipeline_artifacts_checksum_total_count": 5,
"pipeline_artifacts_checksummed_count": 5,
"pipeline_artifacts_checksum_failed_count": 0,
"pipeline_artifacts_synced_count": 5,
"pipeline_artifacts_failed_count": 0,
"pipeline_artifacts_registry_count": 5,
"pipeline_artifacts_verification_total_count": 5,
"pipeline_artifacts_verified_count": 5,
"pipeline_artifacts_verification_failed_count": 0,
"pipeline_artifacts_synced_in_percentage": "100.00%",
"pipeline_artifacts_verified_in_percentage": "100.00%",
}
]
```
......@@ -689,6 +713,18 @@ Example response:
"group_wiki_repositories_verification_failed_count": 0,
"group_wiki_repositories_synced_in_percentage": "100.00%",
"group_wiki_repositories_verified_in_percentage": "100.00%",
"pipeline_artifacts_count": 5,
"pipeline_artifacts_checksum_total_count": 5,
"pipeline_artifacts_checksummed_count": 5,
"pipeline_artifacts_checksum_failed_count": 0,
"pipeline_artifacts_synced_count": 5,
"pipeline_artifacts_failed_count": 0,
"pipeline_artifacts_registry_count": 5,
"pipeline_artifacts_verification_total_count": 5,
"pipeline_artifacts_verified_count": 5,
"pipeline_artifacts_verification_failed_count": 0,
"pipeline_artifacts_synced_in_percentage": "100.00%",
"pipeline_artifacts_verified_in_percentage": "100.00%",
}
```
......
......@@ -3031,6 +3031,7 @@ Represents an external issue.
| `minimumReverificationInterval` | [`Int`](#int) | The interval (in days) in which the repository verification is valid. Once expired, it will be reverified. |
| `name` | [`String`](#string) | The unique identifier for this Geo node. |
| `packageFileRegistries` | [`PackageFileRegistryConnection`](#packagefileregistryconnection) | Package file registries of the GeoNode. |
| `pipelineArtifactRegistries` | [`PipelineArtifactRegistryConnection`](#pipelineartifactregistryconnection) | Find pipeline artifact registries on this Geo node. Available only when feature flag `geo_pipeline_artifact_replication` is enabled. |
| `primary` | [`Boolean`](#boolean) | Indicates whether this Geo node is the primary. |
| `reposMaxCapacity` | [`Int`](#int) | The maximum concurrency of repository backfill for this secondary node. |
| `selectiveSyncNamespaces` | [`NamespaceConnection`](#namespaceconnection) | The namespaces that should be synced, if `selective_sync_type` == `namespaces`. |
......@@ -4642,6 +4643,40 @@ Information about pagination in a connection.
| `yearPipelinesSuccessful` | [`[Int!]`](#int) | Total yearly successful pipeline count. |
| `yearPipelinesTotals` | [`[Int!]`](#int) | Total yearly pipeline count. |
### `PipelineArtifactRegistry`
Represents the Geo sync and verification state of a pipeline artifact.
| Field | Type | Description |
| ----- | ---- | ----------- |
| `createdAt` | [`Time`](#time) | Timestamp when the PipelineArtifactRegistry was created. |
| `id` | [`ID!`](#id) | ID of the PipelineArtifactRegistry. |
| `lastSyncFailure` | [`String`](#string) | Error message during sync of the PipelineArtifactRegistry. |
| `lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the PipelineArtifactRegistry. |
| `pipelineArtifactId` | [`ID!`](#id) | ID of the pipeline artifact. |
| `retryAt` | [`Time`](#time) | Timestamp after which the PipelineArtifactRegistry should be resynced. |
| `retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the PipelineArtifactRegistry. |
| `state` | [`RegistryState`](#registrystate) | Sync state of the PipelineArtifactRegistry. |
### `PipelineArtifactRegistryConnection`
The connection type for PipelineArtifactRegistry.
| Field | Type | Description |
| ----- | ---- | ----------- |
| `edges` | [`[PipelineArtifactRegistryEdge]`](#pipelineartifactregistryedge) | A list of edges. |
| `nodes` | [`[PipelineArtifactRegistry]`](#pipelineartifactregistry) | A list of nodes. |
| `pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. |
### `PipelineArtifactRegistryEdge`
An edge in a connection.
| Field | Type | Description |
| ----- | ---- | ----------- |
| `cursor` | [`String!`](#string) | A cursor for use in pagination. |
| `node` | [`PipelineArtifactRegistry`](#pipelineartifactregistry) | The item at the end of the edge. |
### `PipelineCancelPayload`
Autogenerated return type of PipelineCancel.
......
# frozen_string_literal: true
module Geo
class PipelineArtifactRegistryFinder
include FrameworkRegistryFinder
end
end
# frozen_string_literal: true
module Resolvers
module Geo
class PipelineArtifactRegistriesResolver < BaseResolver
type ::Types::Geo::GeoNodeType.connection_type, null: true
include RegistriesResolver
end
end
end
......@@ -42,6 +42,11 @@ module Types
null: true,
resolver: ::Resolvers::Geo::GroupWikiRepositoryRegistriesResolver,
description: 'Find group wiki repository registries on this Geo node.'
field :pipeline_artifact_registries, ::Types::Geo::PipelineArtifactRegistryType.connection_type,
null: true,
resolver: ::Resolvers::Geo::PipelineArtifactRegistriesResolver,
description: 'Find pipeline artifact registries on this Geo node.',
feature_flag: :geo_pipeline_artifact_replication
end
end
end
# frozen_string_literal: true
module Types
module Geo
# rubocop:disable Graphql/AuthorizeTypes because it is included
class PipelineArtifactRegistryType < BaseObject
include ::Types::Geo::RegistryType
graphql_name 'PipelineArtifactRegistry'
description 'Represents the Geo sync and verification state of a pipeline artifact'
field :pipeline_artifact_id, GraphQL::ID_TYPE, null: false, description: 'ID of the pipeline artifact.'
end
end
end
# frozen_string_literal: true
module EE
module Ci
module Artifactable
extend ActiveSupport::Concern
class_methods do
# @param primary_key_in [Range, Ci::{Pipeline|Job}Artifact] arg to pass to primary_key_in scope
# @return [ActiveRecord::Relation<Ci::{Pipeline|Job}PipelineArtifact>] everything that should be synced to this node, restricted by primary key
def replicables_for_current_secondary(primary_key_in)
node = ::Gitlab::Geo.current_node
primary_key_in(primary_key_in)
.merge(selective_sync_scope(node))
.merge(object_storage_scope(node))
end
def object_storage_scope(node)
return all if node.sync_object_storage?
with_files_stored_locally
end
def selective_sync_scope(node)
return all unless node.selective_sync?
project_id_in(node.projects)
end
end
end
end
end
......@@ -68,28 +68,6 @@ module EE
super
end
# @param primary_key_in [Range, Ci::JobArtifact] arg to pass to primary_key_in scope
# @return [ActiveRecord::Relation<Ci::JobArtifact>] everything that should be synced to this node, restricted by primary key
def replicables_for_current_secondary(primary_key_in)
node = ::Gitlab::Geo.current_node
primary_key_in(primary_key_in)
.merge(selective_sync_scope(node))
.merge(object_storage_scope(node))
end
def object_storage_scope(node)
return all if node.sync_object_storage?
with_files_stored_locally
end
def selective_sync_scope(node)
return all unless node.selective_sync?
project_id_in(node.projects)
end
end
def log_geo_deleted_event
......
# frozen_string_literal: true
module EE
module Ci
module PipelineArtifact
extend ActiveSupport::Concern
prepended do
include ::Gitlab::Geo::ReplicableModel
include ::Gitlab::Geo::VerificationState
with_replicator ::Geo::PipelineArtifactReplicator
end
end
end
end
# frozen_string_literal: true
module Geo
class PipelineArtifactRegistry < Geo::BaseRegistry
include ::Geo::ReplicableRegistry
include ::Geo::VerifiableRegistry
MODEL_CLASS = ::Ci::PipelineArtifact
MODEL_FOREIGN_KEY = :pipeline_artifact_id
belongs_to :pipeline_artifact, class_name: '::Ci::PipelineArtifact'
end
end
# frozen_string_literal: true
module Geo
class PipelineArtifactReplicator < Gitlab::Geo::Replicator
include ::Geo::BlobReplicatorStrategy
extend ::Gitlab::Utils::Override
def self.model
::Ci::PipelineArtifact
end
def carrierwave_uploader
model_record.file
end
override :replication_enabled_by_default?
def self.replication_enabled_by_default?
false
end
override :verification_feature_flag_enabled?
def self.verification_feature_flag_enabled?
# We are adding verification at the same time as replication, so we don't
# need to toggle verification separately from replication. When the
# replication feature flag is off, then verification is also off (see
# `VerifiableReplicator.verification_enabled?`)
true
end
end
end
......@@ -22,6 +22,7 @@ module Geo
Geo::LfsObjectRegistry,
Geo::MergeRequestDiffRegistry,
Geo::PackageFileRegistry,
Geo::PipelineArtifactRegistry,
Geo::ProjectRegistry,
Geo::TerraformStateVersionRegistry,
Geo::UploadRegistry,
......
---
title: Include pipeline artifacts in Geo replication
merge_request: 57741
author:
type: added
......@@ -24,7 +24,8 @@ module Gitlab
::Geo::PackageFileReplicator,
::Geo::TerraformStateVersionReplicator,
::Geo::SnippetRepositoryReplicator,
::Geo::GroupWikiRepositoryReplicator
::Geo::GroupWikiRepositoryReplicator,
::Geo::PipelineArtifactReplicator
].freeze
def self.current_node
......
# frozen_string_literal: true
FactoryBot.define do
factory :geo_pipeline_artifact_registry, class: 'Geo::PipelineArtifactRegistry' do
pipeline_artifact factory: :ci_pipeline_artifact
state { Geo::PipelineArtifactRegistry.state_value(:pending) }
trait :synced do
state { Geo::PipelineArtifactRegistry.state_value(:synced) }
last_synced_at { 5.days.ago }
end
trait :failed do
state { Geo::PipelineArtifactRegistry.state_value(:failed) }
last_synced_at { 1.day.ago }
retry_count { 2 }
last_sync_failure { 'Random error' }
end
trait :started do
state { Geo::PipelineArtifactRegistry.state_value(:started) }
last_synced_at { 1.day.ago }
retry_count { 0 }
end
end
end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::PipelineArtifactRegistryFinder do
it_behaves_like 'a framework registry finder', :geo_pipeline_artifact_registry
end
......@@ -120,6 +120,18 @@
"wikis_checksum_mismatch_count",
"repositories_retrying_verification_count",
"wikis_retrying_verification_count",
"pipeline_artifacts_count",
"pipeline_artifacts_checksum_total_count",
"pipeline_artifacts_checksummed_count",
"pipeline_artifacts_checksum_failed_count",
"pipeline_artifacts_synced_count",
"pipeline_artifacts_failed_count",
"pipeline_artifacts_registry_count",
"pipeline_artifacts_verification_total_count",
"pipeline_artifacts_verified_count",
"pipeline_artifacts_verification_failed_count",
"pipeline_artifacts_synced_in_percentage",
"pipeline_artifacts_verified_in_percentage",
"repositories_checked_count",
"repositories_checked_failed_count",
"repositories_checked_in_percentage",
......@@ -254,6 +266,18 @@
"group_wiki_repositories_verification_total_count": { "type": ["integer", "null"] },
"group_wiki_repositories_verified_count": { "type": ["integer", "null"] },
"group_wiki_repositories_verified_in_percentage": { "type": "string" },
"pipeline_artifacts_count": { "type": ["integer", "null"] },
"pipeline_artifacts_checksummed_count": { "type": ["integer", "null"] },
"pipeline_artifacts_checksum_failed_count": { "type": ["integer", "null"] },
"pipeline_artifacts_checksum_total_count": { "type": ["integer", "null"] },
"pipeline_artifacts_registry_count": { "type": ["integer", "null"] },
"pipeline_artifacts_failed_count": { "type": ["integer", "null"] },
"pipeline_artifacts_synced_count": { "type": ["integer", "null"] },
"pipeline_artifacts_synced_in_percentage": { "type": "string" },
"pipeline_artifacts_verification_failed_count": { "type": ["integer", "null"] },
"pipeline_artifacts_verification_total_count": { "type": ["integer", "null"] },
"pipeline_artifacts_verified_count": { "type": ["integer", "null"] },
"pipeline_artifacts_verified_in_percentage": { "type": "string" },
"repositories_verified_count": { "type": ["integer", "null"] },
"repositories_verification_failed_count": { "type": ["integer", "null"] },
"repositories_verification_total_count": { "type": ["integer", "null"] },
......
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Resolvers::Geo::PipelineArtifactRegistriesResolver do
it_behaves_like 'a Geo registries resolver', :geo_pipeline_artifact_registry
end
......@@ -14,6 +14,7 @@ RSpec.describe GitlabSchema.types['GeoNode'] do
minimum_reverification_interval merge_request_diff_registries
package_file_registries snippet_repository_registries
terraform_state_version_registries group_wiki_repository_registries
pipeline_artifact_registries
]
expect(described_class).to have_graphql_fields(*expected_fields)
......
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe GitlabSchema.types['PipelineArtifactRegistry'] do
it_behaves_like 'a Geo registry type'
it 'has the expected fields (other than those included in RegistryType)' do
expected_fields = %i[pipeline_artifact_id]
expect(described_class).to have_graphql_fields(*expected_fields).at_least
end
end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Ci::PipelineArtifact do
using RSpec::Parameterized::TableSyntax
include EE::GeoHelpers
describe '#replicables_for_current_secondary' do
# Selective sync is configured relative to the pipeline artifact's project.
#
# Permutations of sync_object_storage combined with object-stored-artifacts
# are tested in code, because the logic is simple, and to do it in the table
# would quadruple its size and have too much duplication.
where(:selective_sync_namespaces, :selective_sync_shards, :factory, :project_factory, :include_expectation) do
nil | nil | [:ci_pipeline_artifact] | [:project] | true
# selective sync by shard
nil | :model | [:ci_pipeline_artifact] | [:project] | true
nil | :other | [:ci_pipeline_artifact] | [:project] | false
# selective sync by namespace
:model_parent | nil | [:ci_pipeline_artifact] | [:project] | true
:model_parent_parent | nil | [:ci_pipeline_artifact] | [:project, :in_subgroup] | true
:other | nil | [:ci_pipeline_artifact] | [:project] | false
:other | nil | [:ci_pipeline_artifact] | [:project, :in_subgroup] | false
# expired
nil | nil | [:ci_pipeline_artifact, :expired] | [:project] | true
end
with_them do
subject(:pipeline_artifact_included) { described_class.replicables_for_current_secondary(ci_pipeline_artifact).exists? }
let(:project) { create(*project_factory) } # rubocop: disable Rails/SaveBang
let(:pipeline) { create(:ci_pipeline, project: project) }
let(:node) do
create(:geo_node_with_selective_sync_for,
model: project,
namespaces: selective_sync_namespaces,
shards: selective_sync_shards,
sync_object_storage: sync_object_storage)
end
before do
stub_artifacts_object_storage
stub_current_geo_node(node)
end
context 'when sync object storage is enabled' do
let(:sync_object_storage) { true }
context 'when the pipeline artifact is locally stored' do
let(:ci_pipeline_artifact) { create(*factory, pipeline: pipeline) }
it { is_expected.to eq(include_expectation) }
end
context 'when the pipeline artifact is object stored' do
let(:ci_pipeline_artifact) { create(*factory, :remote_store, pipeline: pipeline) }
it { is_expected.to eq(include_expectation) }
end
end
context 'when sync object storage is disabled' do
let(:sync_object_storage) { false }
context 'when the pipeline artifact is locally stored' do
let(:ci_pipeline_artifact) { create(*factory, pipeline: pipeline) }
it { is_expected.to eq(include_expectation) }
end
context 'when the pipeline artifact is object stored' do
let(:ci_pipeline_artifact) { create(*factory, :remote_store, pipeline: pipeline) }
it { is_expected.to be_falsey }
end
end
end
end
end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::PipelineArtifactRegistry, :geo, type: :model do
let_it_be(:registry) { create(:geo_pipeline_artifact_registry) }
specify 'factory is valid' do
expect(registry).to be_valid
end
include_examples 'a Geo framework registry'
include_examples 'a Geo verifiable registry'
end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::PipelineArtifactReplicator do
let(:model_record) { build(:ci_pipeline_artifact, :with_coverage_report) }
include_examples 'a blob replicator'
it_behaves_like 'a verifiable replicator'
end
......@@ -37,4 +37,11 @@ RSpec.describe 'Gets registries' do
registry_factory: :geo_group_wiki_repository_registry,
registry_foreign_key_field_name: 'groupWikiRepositoryId'
}
it_behaves_like 'gets registries for', {
field_name: 'pipelineArtifactRegistries',
registry_class_name: 'PipelineArtifactRegistry',
registry_factory: :geo_pipeline_artifact_registry,
registry_foreign_key_field_name: 'pipelineArtifactId'
}
end
......@@ -16,12 +16,14 @@ RSpec.describe Geo::RegistryConsistencyService, :geo, :use_clean_rails_memory_st
def model_class_factory_name(registry_class)
default_factory_name = registry_class::MODEL_CLASS.underscore.tr('/', '_').to_sym
{ Geo::DesignRegistry => :project_with_design,
{
Geo::DesignRegistry => :project_with_design,
Geo::MergeRequestDiffRegistry => :external_merge_request_diff,
Geo::PackageFileRegistry => :package_file_with_file,
Geo::SnippetRepositoryRegistry => :snippet_repository,
Geo::TerraformStateVersionRegistry => :terraform_state_version }
.fetch(registry_class, default_factory_name)
Geo::TerraformStateVersionRegistry => :terraform_state_version,
Geo::PipelineArtifactRegistry => :ci_pipeline_artifact
}.fetch(registry_class, default_factory_name)
end
shared_examples 'registry consistency service' do |klass|
......
......@@ -30,6 +30,8 @@ RSpec.shared_examples 'a blob replicator' do
describe '#handle_after_create_commit' do
it 'creates a Geo::Event' do
model_record.save!
expect do
replicator.handle_after_create_commit
end.to change { ::Geo::Event.count }.by(1)
......
......@@ -546,6 +546,8 @@ RSpec.shared_examples 'a verifiable replicator' do
describe 'background backfill' do
it 'verifies model records' do
model_record.verification_pending!
expect do
Geo::VerificationBatchWorker.new.perform(replicator.replicable_name)
end.to change { model_record.reload.verification_succeeded? }.from(false).to(true)
......
......@@ -85,6 +85,7 @@ RSpec.describe Geo::Secondary::RegistryConsistencyWorker, :geo do
merge_request_diff = create(:merge_request_diff, :external)
package_file = create(:conan_package_file, :conan_package)
terraform_state_version = create(:terraform_state_version)
pipeline_artifact = create(:ci_pipeline_artifact)
upload = create(:upload)
expect(Geo::ContainerRepositoryRegistry.where(container_repository_id: container_repository.id).count).to eq(0)
......@@ -93,6 +94,7 @@ RSpec.describe Geo::Secondary::RegistryConsistencyWorker, :geo do
expect(Geo::LfsObjectRegistry.where(lfs_object_id: lfs_object.id).count).to eq(0)
expect(Geo::MergeRequestDiffRegistry.where(merge_request_diff_id: merge_request_diff.id).count).to eq(0)
expect(Geo::PackageFileRegistry.where(package_file_id: package_file.id).count).to eq(0)
expect(Geo::PipelineArtifactRegistry.where(pipeline_artifact_id: pipeline_artifact.id).count).to eq(0)
expect(Geo::ProjectRegistry.where(project_id: project.id).count).to eq(0)
expect(Geo::TerraformStateVersionRegistry.where(terraform_state_version_id: terraform_state_version.id).count).to eq(0)
expect(Geo::UploadRegistry.where(file_id: upload.id).count).to eq(0)
......@@ -105,6 +107,7 @@ RSpec.describe Geo::Secondary::RegistryConsistencyWorker, :geo do
expect(Geo::LfsObjectRegistry.where(lfs_object_id: lfs_object.id).count).to eq(1)
expect(Geo::MergeRequestDiffRegistry.where(merge_request_diff_id: merge_request_diff.id).count).to eq(1)
expect(Geo::PackageFileRegistry.where(package_file_id: package_file.id).count).to eq(1)
expect(Geo::PipelineArtifactRegistry.where(pipeline_artifact_id: pipeline_artifact.id).count).to eq(1)
expect(Geo::ProjectRegistry.where(project_id: project.id).count).to eq(1)
expect(Geo::TerraformStateVersionRegistry.where(terraform_state_version_id: terraform_state_version.id).count).to eq(1)
expect(Geo::UploadRegistry.where(file_id: upload.id).count).to eq(1)
......
......@@ -13,6 +13,22 @@ FactoryBot.define do
Rails.root.join('spec/fixtures/pipeline_artifacts/code_coverage.json'), 'application/json')
end
trait :checksummed do
verification_checksum { 'abc' }
end
trait :checksum_failure do
verification_failure { 'Could not calculate the checksum' }
end
trait :expired do
expire_at { Date.yesterday }
end
trait :remote_store do
file_store { ::ObjectStorage::Store::REMOTE}
end
trait :with_coverage_report do
file_type { :code_coverage }
......
......@@ -32,7 +32,8 @@ RSpec.describe 'factories' do
[:project_member, :blocked],
[:project, :remote_mirror],
[:remote_mirror, :ssh],
[:user_preference, :only_comments]
[:user_preference, :only_comments],
[:ci_pipeline_artifact, :remote_store]
]
end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment