Commit 4c768b1b authored by Michael Kozono's avatar Michael Kozono

Merge branch '270449-geo-uploads-to-ssf-no-verification' into 'master'

Geo: Migrate Uploads replication to SSF (no verification)

See merge request gitlab-org/gitlab!68278
parents 0b219734 c6597b43
...@@ -268,6 +268,10 @@ configuration option in `gitlab.yml`. These metrics are served from the ...@@ -268,6 +268,10 @@ configuration option in `gitlab.yml`. These metrics are served from the
| `destroyed_job_artifacts_count_total` | Counter | 13.6 | Number of destroyed expired job artifacts | | | `destroyed_job_artifacts_count_total` | Counter | 13.6 | Number of destroyed expired job artifacts | |
| `destroyed_pipeline_artifacts_count_total` | Counter | 13.8 | Number of destroyed expired pipeline artifacts | | | `destroyed_pipeline_artifacts_count_total` | Counter | 13.8 | Number of destroyed expired pipeline artifacts | |
| `gitlab_optimistic_locking_retries` | Histogram | 13.10 | Number of retry attempts to execute optimistic retry lock | | | `gitlab_optimistic_locking_retries` | Histogram | 13.10 | Number of retry attempts to execute optimistic retry lock | |
| `geo_uploads` | Gauge | 14.1 | Number of uploads on primary | `url` |
| `geo_uploads_synced` | Gauge | 14.1 | Number of uploads synced on secondary | `url` |
| `geo_uploads_failed` | Gauge | 14.1 | Number of syncable uploads failed to sync on secondary | `url` |
| `geo_uploads_registry` | Gauge | 14.1 | Number of uploads in the registry | `url` |
## Database load balancing metrics **(PREMIUM SELF)** ## Database load balancing metrics **(PREMIUM SELF)**
......
...@@ -453,6 +453,11 @@ Example response: ...@@ -453,6 +453,11 @@ Example response:
"pipeline_artifacts_verification_failed_count": null, "pipeline_artifacts_verification_failed_count": null,
"pipeline_artifacts_synced_in_percentage": "0.00%", "pipeline_artifacts_synced_in_percentage": "0.00%",
"pipeline_artifacts_verified_in_percentage": "0.00%", "pipeline_artifacts_verified_in_percentage": "0.00%",
"uploads_count": 5,
"uploads_synced_count": null,
"uploads_failed_count": 0,
"uploads_registry_count": null,
"uploads_synced_in_percentage": "0.00%",
}, },
{ {
"geo_node_id": 2, "geo_node_id": 2,
...@@ -595,6 +600,11 @@ Example response: ...@@ -595,6 +600,11 @@ Example response:
"pipeline_artifacts_verification_failed_count": 0, "pipeline_artifacts_verification_failed_count": 0,
"pipeline_artifacts_synced_in_percentage": "100.00%", "pipeline_artifacts_synced_in_percentage": "100.00%",
"pipeline_artifacts_verified_in_percentage": "100.00%", "pipeline_artifacts_verified_in_percentage": "100.00%",
"uploads_count": 5,
"uploads_synced_count": null,
"uploads_failed_count": 0,
"uploads_registry_count": null,
"uploads_synced_in_percentage": "0.00%",
} }
] ]
``` ```
...@@ -734,6 +744,11 @@ Example response: ...@@ -734,6 +744,11 @@ Example response:
"pipeline_artifacts_verification_failed_count": 0, "pipeline_artifacts_verification_failed_count": 0,
"pipeline_artifacts_synced_in_percentage": "100.00%", "pipeline_artifacts_synced_in_percentage": "100.00%",
"pipeline_artifacts_verified_in_percentage": "100.00%", "pipeline_artifacts_verified_in_percentage": "100.00%",
"uploads_count": 5,
"uploads_synced_count": null,
"uploads_failed_count": 0,
"uploads_registry_count": null,
"uploads_synced_in_percentage": "0.00%",
} }
``` ```
......
...@@ -7443,6 +7443,29 @@ The edge type for [`TreeEntry`](#treeentry). ...@@ -7443,6 +7443,29 @@ The edge type for [`TreeEntry`](#treeentry).
| <a id="treeentryedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="treeentryedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. |
| <a id="treeentryedgenode"></a>`node` | [`TreeEntry`](#treeentry) | The item at the end of the edge. | | <a id="treeentryedgenode"></a>`node` | [`TreeEntry`](#treeentry) | The item at the end of the edge. |
#### `UploadRegistryConnection`
The connection type for [`UploadRegistry`](#uploadregistry).
##### Fields
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="uploadregistryconnectionedges"></a>`edges` | [`[UploadRegistryEdge]`](#uploadregistryedge) | A list of edges. |
| <a id="uploadregistryconnectionnodes"></a>`nodes` | [`[UploadRegistry]`](#uploadregistry) | A list of nodes. |
| <a id="uploadregistryconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. |
#### `UploadRegistryEdge`
The edge type for [`UploadRegistry`](#uploadregistry).
##### Fields
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="uploadregistryedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. |
| <a id="uploadregistryedgenode"></a>`node` | [`UploadRegistry`](#uploadregistry) | The item at the end of the edge. |
#### `UsageTrendsMeasurementConnection` #### `UsageTrendsMeasurementConnection`
The connection type for [`UsageTrendsMeasurement`](#usagetrendsmeasurement). The connection type for [`UsageTrendsMeasurement`](#usagetrendsmeasurement).
...@@ -9951,6 +9974,22 @@ four standard [pagination arguments](#connection-pagination-arguments): ...@@ -9951,6 +9974,22 @@ four standard [pagination arguments](#connection-pagination-arguments):
| ---- | ---- | ----------- | | ---- | ---- | ----------- |
| <a id="geonodeterraformstateversionregistriesids"></a>`ids` | [`[ID!]`](#id) | Filters registries by their ID. | | <a id="geonodeterraformstateversionregistriesids"></a>`ids` | [`[ID!]`](#id) | Filters registries by their ID. |
##### `GeoNode.uploadRegistries`
Find Upload registries on this Geo node Available only when feature flag `geo_upload_replication` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice.
Returns [`UploadRegistryConnection`](#uploadregistryconnection).
This field returns a [connection](#connections). It accepts the
four standard [pagination arguments](#connection-pagination-arguments):
`before: String`, `after: String`, `first: Int`, `last: Int`.
###### Arguments
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="geonodeuploadregistriesids"></a>`ids` | [`[ID!]`](#id) | Filters registries by their ID. |
### `GrafanaIntegration` ### `GrafanaIntegration`
#### Fields #### Fields
...@@ -14347,6 +14386,23 @@ Represents a directory. ...@@ -14347,6 +14386,23 @@ Represents a directory.
| <a id="treeentrywebpath"></a>`webPath` | [`String`](#string) | Web path for the tree entry (directory). | | <a id="treeentrywebpath"></a>`webPath` | [`String`](#string) | Web path for the tree entry (directory). |
| <a id="treeentryweburl"></a>`webUrl` | [`String`](#string) | Web URL for the tree entry (directory). | | <a id="treeentryweburl"></a>`webUrl` | [`String`](#string) | Web URL for the tree entry (directory). |
### `UploadRegistry`
Represents the Geo replication and verification state of an upload.
#### Fields
| Name | Type | Description |
| ---- | ---- | ----------- |
| <a id="uploadregistrycreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp when the UploadRegistry was created. |
| <a id="uploadregistryfileid"></a>`fileId` | [`ID!`](#id) | ID of the Upload. |
| <a id="uploadregistryid"></a>`id` | [`ID!`](#id) | ID of the UploadRegistry. |
| <a id="uploadregistrylastsyncfailure"></a>`lastSyncFailure` | [`String`](#string) | Error message during sync of the UploadRegistry. |
| <a id="uploadregistrylastsyncedat"></a>`lastSyncedAt` | [`Time`](#time) | Timestamp of the most recent successful sync of the UploadRegistry. |
| <a id="uploadregistryretryat"></a>`retryAt` | [`Time`](#time) | Timestamp after which the UploadRegistry should be resynced. |
| <a id="uploadregistryretrycount"></a>`retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the UploadRegistry. |
| <a id="uploadregistrystate"></a>`state` | [`RegistryState`](#registrystate) | Sync state of the UploadRegistry. |
### `UsageTrendsMeasurement` ### `UsageTrendsMeasurement`
Represents a recorded measurement (object count) for the Admins. Represents a recorded measurement (object count) for the Admins.
......
# frozen_string_literal: true # frozen_string_literal: true
module Geo module Geo
class AttachmentRegistryFinder < FileRegistryFinder class AttachmentLegacyRegistryFinder < FileRegistryFinder
def registry_class def registry_class
Geo::UploadRegistry Geo::UploadRegistry
end end
......
# frozen_string_literal: true
module Geo
class UploadRegistryFinder
include FrameworkRegistryFinder
end
end
# frozen_string_literal: true
module Resolvers
module Geo
class UploadRegistriesResolver < BaseResolver
type ::Types::Geo::GeoNodeType.connection_type, null: true
include RegistriesResolver
end
end
end
...@@ -54,6 +54,11 @@ module Types ...@@ -54,6 +54,11 @@ module Types
null: true, null: true,
resolver: ::Resolvers::Geo::PagesDeploymentRegistriesResolver, resolver: ::Resolvers::Geo::PagesDeploymentRegistriesResolver,
description: 'Find Pages Deployment registries on this Geo node' description: 'Find Pages Deployment registries on this Geo node'
field :upload_registries, ::Types::Geo::UploadRegistryType.connection_type,
null: true,
resolver: ::Resolvers::Geo::UploadRegistriesResolver,
description: 'Find Upload registries on this Geo node',
feature_flag: :geo_upload_replication
end end
end end
end end
# frozen_string_literal: true
module Types
module Geo
# rubocop:disable Graphql/AuthorizeTypes because it is included
class UploadRegistryType < BaseObject
include ::Types::Geo::RegistryType
graphql_name 'UploadRegistry'
description 'Represents the Geo replication and verification state of an upload.'
field :file_id, GraphQL::Types::ID, null: false, description: 'ID of the Upload.'
end
end
end
...@@ -197,15 +197,6 @@ module EE ...@@ -197,15 +197,6 @@ module EE
name: 'wiki', name: 'wiki',
name_plural: 'wikis' name_plural: 'wikis'
}, },
{
data_type: 'blob',
data_type_title: _('File'),
title: _('Upload'),
title_plural: _('Uploads'),
name: 'attachment',
name_plural: 'attachments',
secondary_view: true
},
{ {
data_type: 'blob', data_type: 'blob',
data_type_title: _('File'), data_type_title: _('File'),
...@@ -233,6 +224,18 @@ module EE ...@@ -233,6 +224,18 @@ module EE
} }
] ]
if ::Geo::UploadReplicator.disabled?
replicable_types.insert(2, {
data_type: 'blob',
data_type_title: _('File'),
title: _('Upload'),
title_plural: _('Uploads'),
name: 'attachment',
name_plural: 'attachments',
secondary_view: true
})
end
# Adds all the SSF Data Types automatically # Adds all the SSF Data Types automatically
enabled_replicator_classes.each do |replicator_class| enabled_replicator_classes.each do |replicator_class|
replicable_types.push( replicable_types.push(
......
...@@ -55,7 +55,7 @@ module Geo ...@@ -55,7 +55,7 @@ module Geo
raise NotImplementedError raise NotImplementedError
end end
# Return the absolute path to locally stored package file # Return the absolute path to locally stored file
# #
# @return [String] File path # @return [String] File path
def blob_path def blob_path
...@@ -76,7 +76,7 @@ module Geo ...@@ -76,7 +76,7 @@ module Geo
def calculate_checksum def calculate_checksum
raise 'File is not checksummable' unless checksummable? raise 'File is not checksummable' unless checksummable?
model.hexdigest(carrierwave_uploader.path) model.hexdigest(blob_path)
end end
# Returns whether the file exists on disk or in remote storage # Returns whether the file exists on disk or in remote storage
......
...@@ -7,7 +7,7 @@ module EE ...@@ -7,7 +7,7 @@ module EE
prepended do prepended do
include ::Gitlab::Geo::ReplicableModel include ::Gitlab::Geo::ReplicableModel
with_replicator Geo::PagesDeploymentReplicator with_replicator ::Geo::PagesDeploymentReplicator
end end
class_methods do class_methods do
......
...@@ -10,6 +10,9 @@ module EE ...@@ -10,6 +10,9 @@ module EE
prepended do prepended do
include ::Gitlab::SQL::Pattern include ::Gitlab::SQL::Pattern
include ::Gitlab::Geo::ReplicableModel
with_replicator ::Geo::UploadReplicator
after_destroy :log_geo_deleted_event after_destroy :log_geo_deleted_event
......
...@@ -2,6 +2,9 @@ ...@@ -2,6 +2,9 @@
class Geo::UploadRegistry < Geo::BaseRegistry class Geo::UploadRegistry < Geo::BaseRegistry
include Geo::Syncable include Geo::Syncable
include ::Geo::ReplicableRegistry
extend ::Gitlab::Utils::Override
MODEL_CLASS = ::Upload MODEL_CLASS = ::Upload
MODEL_FOREIGN_KEY = :file_id MODEL_FOREIGN_KEY = :file_id
...@@ -51,7 +54,7 @@ class Geo::UploadRegistry < Geo::BaseRegistry ...@@ -51,7 +54,7 @@ class Geo::UploadRegistry < Geo::BaseRegistry
# If false, RegistryConsistencyService will frequently check the end of the # If false, RegistryConsistencyService will frequently check the end of the
# table to quickly handle new replicables. # table to quickly handle new replicables.
def self.has_create_events? def self.has_create_events?
false ::Geo::UploadReplicator.enabled?
end end
def self.insert_for_model_ids(attrs) def self.insert_for_model_ids(attrs)
...@@ -109,4 +112,43 @@ class Geo::UploadRegistry < Geo::BaseRegistry ...@@ -109,4 +112,43 @@ class Geo::UploadRegistry < Geo::BaseRegistry
:failed :failed
end end
# TODO Remove this when enabling geo_upload_registry by default
# https://gitlab.com/gitlab-org/gitlab/-/issues/340617
override :registry_consistency_worker_enabled?
def self.registry_consistency_worker_enabled?
true
end
def self.failed
if ::Geo::UploadReplicator.enabled?
with_state(:failed)
else
where(success: false).where.not(retry_count: nil)
end
end
def self.never_attempted_sync
if ::Geo::UploadReplicator.enabled?
pending.where(last_synced_at: nil)
else
where(success: false, retry_count: nil)
end
end
def self.retry_due
if ::Geo::UploadReplicator.enabled?
where(arel_table[:retry_at].eq(nil).or(arel_table[:retry_at].lt(Time.current)))
else
where('retry_at is NULL OR retry_at < ?', Time.current)
end
end
def self.synced
if ::Geo::UploadReplicator.enabled?
with_state(:synced).or(where(success: true))
else
where(success: true)
end
end
end end
...@@ -595,7 +595,7 @@ class GeoNodeStatus < ApplicationRecord ...@@ -595,7 +595,7 @@ class GeoNodeStatus < ApplicationRecord
end end
def attachments_finder def attachments_finder
@attachments_finder ||= Geo::AttachmentRegistryFinder.new @attachments_finder ||= Geo::AttachmentLegacyRegistryFinder.new
end end
def job_artifacts_finder def job_artifacts_finder
......
# frozen_string_literal: true
module Geo
class UploadReplicator < Gitlab::Geo::Replicator
include ::Geo::BlobReplicatorStrategy
extend ::Gitlab::Utils::Override
def self.model
::Upload
end
def self.replication_enabled_by_default?
false
end
def carrierwave_uploader
model_record.retrieve_uploader
end
# TODO: This method can be removed as part of
# https://gitlab.com/gitlab-org/gitlab/-/issues/340617
override :registry
def registry
super.tap do |record|
# We don't really need this value for SSF, it's only needed to make
# new registry records valid for legacy code in case of disabling the feature.
record.file_type ||= model_record.uploader.delete_suffix("Uploader").underscore
end
end
end
end
...@@ -10,10 +10,11 @@ ...@@ -10,10 +10,11 @@
= link_to admin_geo_projects_path, title: _('Projects') do = link_to admin_geo_projects_path, title: _('Projects') do
%span %span
= _('Projects') = _('Projects')
= nav_link(path: 'admin/geo/uploads#index', html_options: { class: 'gl-pr-2' }) do - if ::Geo::UploadReplicator.disabled?
= link_to admin_geo_uploads_path, title: _('Uploads') do = nav_link(path: 'admin/geo/uploads#index', html_options: { class: 'gl-pr-2' }) do
%span = link_to admin_geo_uploads_path, title: _('Uploads') do
= _('Uploads') %span
= _('Uploads')
= nav_link(path: 'admin/geo/designs#index', html_options: { class: 'gl-pr-2' }) do = nav_link(path: 'admin/geo/designs#index', html_options: { class: 'gl-pr-2' }) do
= link_to admin_geo_designs_path, title: _('Designs') do = link_to admin_geo_designs_path, title: _('Designs') do
%span %span
......
...@@ -71,10 +71,13 @@ module Geo ...@@ -71,10 +71,13 @@ module Geo
end end
def job_finders def job_finders
[ job_finders = [Geo::FileDownloadDispatchWorker::JobArtifactJobFinder.new(scheduled_file_ids(:job_artifact))]
Geo::FileDownloadDispatchWorker::AttachmentJobFinder.new(scheduled_file_ids(Gitlab::Geo::Replication::USER_UPLOADS_OBJECT_TYPES)),
Geo::FileDownloadDispatchWorker::JobArtifactJobFinder.new(scheduled_file_ids(:job_artifact)) if ::Geo::UploadReplicator.disabled?
] job_finders << Geo::FileDownloadDispatchWorker::AttachmentJobFinder.new(scheduled_file_ids(Gitlab::Geo::Replication::USER_UPLOADS_OBJECT_TYPES))
end
job_finders
end end
def scheduled_file_ids(file_types) def scheduled_file_ids(file_types)
......
...@@ -6,7 +6,7 @@ module Geo ...@@ -6,7 +6,7 @@ module Geo
EXCEPT_RESOURCE_IDS_KEY = :except_ids EXCEPT_RESOURCE_IDS_KEY = :except_ids
def registry_finder def registry_finder
@registry_finder ||= Geo::AttachmentRegistryFinder.new @registry_finder ||= Geo::AttachmentLegacyRegistryFinder.new
end end
private private
......
---
name: geo_upload_replication
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68278
rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/340617
milestone: '14.4'
type: development
group: group::geo
default_enabled: false
...@@ -47,7 +47,6 @@ namespace :admin do ...@@ -47,7 +47,6 @@ namespace :admin do
# Old Routes Replaced in 13.0 # Old Routes Replaced in 13.0
get '/projects', to: redirect(path: 'admin/geo/replication/projects') get '/projects', to: redirect(path: 'admin/geo/replication/projects')
get '/uploads', to: redirect(path: 'admin/geo/replication/uploads')
get '/designs', to: redirect(path: 'admin/geo/replication/designs') get '/designs', to: redirect(path: 'admin/geo/replication/designs')
resources :nodes, only: [:index, :create, :new, :edit, :update] resources :nodes, only: [:index, :create, :new, :edit, :update]
...@@ -70,7 +69,7 @@ namespace :admin do ...@@ -70,7 +69,7 @@ namespace :admin do
resources :designs, only: [:index] resources :designs, only: [:index]
resources :uploads, only: [:index, :destroy] resources :uploads, only: [:index, :destroy], path: 'legacy-uploads'
get '/:replicable_name_plural', to: 'replicables#index', as: 'replicables' get '/:replicable_name_plural', to: 'replicables#index', as: 'replicables'
end end
......
# frozen_string_literal: true
class PrepareFileRegistryForSsf < ActiveRecord::Migration[6.1]
def change
change_column_default :file_registry, :retry_count, from: nil, to: 0
add_column :file_registry, :state, :integer, null: false, limit: 2, default: 0
add_column :file_registry, :last_synced_at, :datetime_with_timezone
add_column :file_registry, :last_sync_failure, :string, limit: 255 # rubocop:disable Migration/PreventStrings see https://gitlab.com/gitlab-org/gitlab/-/issues/323806
end
end
# frozen_string_literal: true
class FixStateColumnInFileRegistry < ActiveRecord::Migration[6.1]
include Gitlab::Database::MigrationHelpers
disable_ddl_transaction!
# The following cop is disabled because of https://gitlab.com/gitlab-org/gitlab/issues/33470
# rubocop:disable Migration/UpdateColumnInBatches
def up
update_column_in_batches(:file_registry, :state, 2) do |table, query|
query.where(table[:success].eq(true)) # rubocop:disable CodeReuse/ActiveRecord
end
end
# rubocop:enable Migration/UpdateColumnInBatches
def down
# no-op
end
end
...@@ -54,9 +54,12 @@ ActiveRecord::Schema.define(version: 2021_08_20_152707) do ...@@ -54,9 +54,12 @@ ActiveRecord::Schema.define(version: 2021_08_20_152707) do
t.string "sha256" t.string "sha256"
t.datetime "created_at", null: false t.datetime "created_at", null: false
t.boolean "success", default: false, null: false t.boolean "success", default: false, null: false
t.integer "retry_count" t.integer "retry_count", default: 0
t.datetime "retry_at" t.datetime "retry_at"
t.boolean "missing_on_primary", default: false, null: false t.boolean "missing_on_primary", default: false, null: false
t.integer "state", limit: 2, default: 0, null: false
t.datetime_with_timezone "last_synced_at"
t.string "last_sync_failure", limit: 255
t.index ["file_type", "file_id"], name: "index_file_registry_on_file_type_and_file_id", unique: true t.index ["file_type", "file_id"], name: "index_file_registry_on_file_type_and_file_id", unique: true
t.index ["file_type"], name: "index_file_registry_on_file_type" t.index ["file_type"], name: "index_file_registry_on_file_type"
t.index ["retry_at"], name: "index_file_registry_on_retry_at" t.index ["retry_at"], name: "index_file_registry_on_retry_at"
......
...@@ -35,7 +35,7 @@ module EE ...@@ -35,7 +35,7 @@ module EE
expose :db_replication_lag_seconds expose :db_replication_lag_seconds
expose :attachments_replication_enabled expose :attachments_replication_enabled, if: -> (*) { ::Geo::UploadReplicator.disabled? }
expose :job_artifacts_replication_enabled expose :job_artifacts_replication_enabled
expose :container_repositories_replication_enabled expose :container_repositories_replication_enabled
expose :design_repositories_replication_enabled expose :design_repositories_replication_enabled
......
...@@ -27,7 +27,8 @@ module Gitlab ...@@ -27,7 +27,8 @@ module Gitlab
::Geo::SnippetRepositoryReplicator, ::Geo::SnippetRepositoryReplicator,
::Geo::GroupWikiRepositoryReplicator, ::Geo::GroupWikiRepositoryReplicator,
::Geo::PipelineArtifactReplicator, ::Geo::PipelineArtifactReplicator,
::Geo::PagesDeploymentReplicator ::Geo::PagesDeploymentReplicator,
::Geo::UploadReplicator
].freeze ].freeze
def self.current_node def self.current_node
......
...@@ -253,6 +253,8 @@ module Gitlab ...@@ -253,6 +253,8 @@ module Gitlab
end end
def print_attachments_status def print_attachments_status
return if ::Geo::UploadReplicator.enabled?
print 'Attachments: '.rjust(GEO_STATUS_COLUMN_WIDTH) print 'Attachments: '.rjust(GEO_STATUS_COLUMN_WIDTH)
show_failed_value(current_node_status.attachments_failed_count) show_failed_value(current_node_status.attachments_failed_count)
print "#{current_node_status.attachments_synced_count}/#{current_node_status.attachments_count} " print "#{current_node_status.attachments_synced_count}/#{current_node_status.attachments_count} "
......
...@@ -159,6 +159,10 @@ module Gitlab ...@@ -159,6 +159,10 @@ module Gitlab
default_enabled: replication_enabled_by_default?) default_enabled: replication_enabled_by_default?)
end end
def self.disabled?
!enabled?
end
# Replication is set behind a feature flag, which is enabled by default. # Replication is set behind a feature flag, which is enabled by default.
# If you want it disabled by default, override this method. # If you want it disabled by default, override this method.
def self.replication_enabled_by_default? def self.replication_enabled_by_default?
......
...@@ -7,15 +7,17 @@ RSpec.describe Admin::Geo::UploadsController, :geo do ...@@ -7,15 +7,17 @@ RSpec.describe Admin::Geo::UploadsController, :geo do
let_it_be(:admin) { create(:admin) } let_it_be(:admin) { create(:admin) }
let_it_be(:secondary) { create(:geo_node) } let_it_be(:secondary) { create(:geo_node) }
let_it_be(:synced_registry) { create(:geo_upload_registry, :with_file, :attachment, success: true) } let_it_be(:synced_registry) { create(:geo_upload_legacy_registry, :with_file, :attachment, success: true) }
let_it_be(:failed_registry) { create(:geo_upload_registry, :failed) } let_it_be(:failed_registry) { create(:geo_upload_legacy_registry, :failed) }
let_it_be(:never_registry) { create(:geo_upload_registry, :failed, retry_count: nil) } let_it_be(:never_registry) { create(:geo_upload_legacy_registry, :failed, retry_count: nil) }
def css_id(registry) def css_id(registry)
"#upload-#{registry.id}-header" "#upload-#{registry.id}-header"
end end
before do before do
stub_feature_flags(geo_upload_replication: false)
sign_in(admin) sign_in(admin)
end end
...@@ -94,7 +96,7 @@ RSpec.describe Admin::Geo::UploadsController, :geo do ...@@ -94,7 +96,7 @@ RSpec.describe Admin::Geo::UploadsController, :geo do
subject { delete :destroy, params: { id: registry } } subject { delete :destroy, params: { id: registry } }
it_behaves_like 'license required' do it_behaves_like 'license required' do
let(:registry) { create(:geo_upload_registry) } let(:registry) { create(:geo_upload_legacy_registry) }
end end
context 'with a valid license' do context 'with a valid license' do
...@@ -103,7 +105,7 @@ RSpec.describe Admin::Geo::UploadsController, :geo do ...@@ -103,7 +105,7 @@ RSpec.describe Admin::Geo::UploadsController, :geo do
end end
context 'with an orphaned registry' do context 'with an orphaned registry' do
let(:registry) { create(:geo_upload_registry, success: true) } let(:registry) { create(:geo_upload_legacy_registry, success: true) }
it 'removes the registry' do it 'removes the registry' do
registry.update_column(:file_id, -1) registry.update_column(:file_id, -1)
...@@ -115,7 +117,7 @@ RSpec.describe Admin::Geo::UploadsController, :geo do ...@@ -115,7 +117,7 @@ RSpec.describe Admin::Geo::UploadsController, :geo do
end end
context 'with a regular registry' do context 'with a regular registry' do
let(:registry) { create(:geo_upload_registry, :avatar, :with_file, success: true) } let(:registry) { create(:geo_upload_legacy_registry, :avatar, :with_file, success: true) }
it 'does not delete the registry and gives an error' do it 'does not delete the registry and gives an error' do
expect(subject).to redirect_to(admin_geo_uploads_path) expect(subject).to redirect_to(admin_geo_uploads_path)
......
# frozen_string_literal: true # frozen_string_literal: true
FactoryBot.define do FactoryBot.define do
factory :geo_upload_registry, class: 'Geo::UploadRegistry' do factory :geo_upload_legacy_registry, class: 'Geo::UploadRegistry' do
sequence(:file_id) sequence(:file_id)
file_type { :file } file_type { :file }
success { true } success { true }
...@@ -40,3 +40,30 @@ FactoryBot.define do ...@@ -40,3 +40,30 @@ FactoryBot.define do
end end
end end
end end
FactoryBot.define do
factory :geo_upload_registry, class: 'Geo::UploadRegistry' do
association(:upload, :with_file)
sequence(:file_id)
file_type { :file }
state { Geo::UploadRegistry.state_value(:pending) }
trait :synced do
state { Geo::UploadRegistry.state_value(:synced) }
last_synced_at { 5.days.ago }
end
trait :failed do
state { Geo::UploadRegistry.state_value(:failed) }
last_synced_at { 1.day.ago }
retry_count { 2 }
last_sync_failure { 'Random error' }
end
trait :started do
state { Geo::UploadRegistry.state_value(:started) }
last_synced_at { 1.day.ago }
retry_count { 0 }
end
end
end
...@@ -33,7 +33,11 @@ RSpec.describe 'admin Geo Replication Nav', :js, :geo do ...@@ -33,7 +33,11 @@ RSpec.describe 'admin Geo Replication Nav', :js, :geo do
end end
end end
describe 'visit admin/geo/replication/uploads' do describe 'visit admin/geo/replication/legacy-uploads' do
before do
stub_feature_flags(geo_upload_replication: false)
end
it_behaves_like 'active sidebar link', 'Uploads' do it_behaves_like 'active sidebar link', 'Uploads' do
let(:path) { admin_geo_uploads_path } let(:path) { admin_geo_uploads_path }
end end
......
...@@ -4,10 +4,12 @@ require 'spec_helper' ...@@ -4,10 +4,12 @@ require 'spec_helper'
RSpec.describe 'admin Geo Uploads', :js, :geo do RSpec.describe 'admin Geo Uploads', :js, :geo do
let!(:geo_node) { create(:geo_node) } let!(:geo_node) { create(:geo_node) }
let!(:synced_registry) { create(:geo_upload_registry, :with_file, :attachment, success: true) } let!(:synced_registry) { create(:geo_upload_legacy_registry, :with_file, :attachment, success: true) }
before do before do
allow(Gitlab::Geo).to receive(:license_allows?).and_return(true) allow(Gitlab::Geo).to receive(:license_allows?).and_return(true)
stub_feature_flags(geo_upload_replication: false)
admin = create(:admin) admin = create(:admin)
sign_in(admin) sign_in(admin)
gitlab_enable_admin_mode_sign_in(admin) gitlab_enable_admin_mode_sign_in(admin)
......
...@@ -2,7 +2,11 @@ ...@@ -2,7 +2,11 @@
require 'spec_helper' require 'spec_helper'
RSpec.describe Geo::AttachmentRegistryFinder, :geo do RSpec.describe Geo::AttachmentLegacyRegistryFinder, :geo do
before do
stub_feature_flags(geo_upload_replication: false )
end
it_behaves_like 'a file registry finder' do it_behaves_like 'a file registry finder' do
let_it_be(:project) { create(:project) } let_it_be(:project) { create(:project) }
...@@ -16,13 +20,13 @@ RSpec.describe Geo::AttachmentRegistryFinder, :geo do ...@@ -16,13 +20,13 @@ RSpec.describe Geo::AttachmentRegistryFinder, :geo do
let_it_be(:replicable_8) { create(:upload, :object_storage, model: project) } let_it_be(:replicable_8) { create(:upload, :object_storage, model: project) }
let_it_be(:replicable_9) { create(:upload, :object_storage, model: project) } let_it_be(:replicable_9) { create(:upload, :object_storage, model: project) }
let_it_be(:registry_1) { create(:geo_upload_registry, :attachment, :failed, file_id: replicable_1.id) } let_it_be(:registry_1) { create(:geo_upload_legacy_registry, :attachment, :failed, file_id: replicable_1.id) }
let_it_be(:registry_2) { create(:geo_upload_registry, :attachment, file_id: replicable_2.id, missing_on_primary: true) } let_it_be(:registry_2) { create(:geo_upload_legacy_registry, :attachment, file_id: replicable_2.id, missing_on_primary: true) }
let_it_be(:registry_3) { create(:geo_upload_registry, :attachment, :never_synced, file_id: replicable_3.id) } let_it_be(:registry_3) { create(:geo_upload_legacy_registry, :attachment, :never_synced, file_id: replicable_3.id) }
let_it_be(:registry_4) { create(:geo_upload_registry, :attachment, :failed, file_id: replicable_4.id) } let_it_be(:registry_4) { create(:geo_upload_legacy_registry, :attachment, :failed, file_id: replicable_4.id) }
let_it_be(:registry_5) { create(:geo_upload_registry, :attachment, file_id: replicable_5.id, missing_on_primary: true, retry_at: 1.day.ago) } let_it_be(:registry_5) { create(:geo_upload_legacy_registry, :attachment, file_id: replicable_5.id, missing_on_primary: true, retry_at: 1.day.ago) }
let_it_be(:registry_6) { create(:geo_upload_registry, :attachment, :failed, file_id: replicable_6.id) } let_it_be(:registry_6) { create(:geo_upload_legacy_registry, :attachment, :failed, file_id: replicable_6.id) }
let_it_be(:registry_7) { create(:geo_upload_registry, :attachment, :failed, file_id: replicable_7.id, missing_on_primary: true) } let_it_be(:registry_7) { create(:geo_upload_legacy_registry, :attachment, :failed, file_id: replicable_7.id, missing_on_primary: true) }
let_it_be(:registry_8) { create(:geo_upload_registry, :attachment, :never_synced, file_id: replicable_8.id) } let_it_be(:registry_8) { create(:geo_upload_legacy_registry, :attachment, :never_synced, file_id: replicable_8.id) }
end end
end end
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::UploadRegistryFinder do
it_behaves_like 'a framework registry finder', :geo_upload_registry
end
...@@ -157,6 +157,18 @@ ...@@ -157,6 +157,18 @@
"replication_slots_used_count", "replication_slots_used_count",
"replication_slots_used_in_percentage", "replication_slots_used_in_percentage",
"replication_slots_max_retained_wal_bytes", "replication_slots_max_retained_wal_bytes",
"uploads_count",
"uploads_synced_count",
"uploads_failed_count",
"uploads_registry_count",
"uploads_synced_in_percentage",
"uploads_checksum_total_count",
"uploads_checksummed_count",
"uploads_checksum_failed_count",
"uploads_verification_failed_count",
"uploads_verification_total_count",
"uploads_verified_count",
"uploads_verified_in_percentage",
"git_fetch_event_count_weekly", "git_fetch_event_count_weekly",
"git_push_event_count_weekly", "git_push_event_count_weekly",
"last_event_id", "last_event_id",
...@@ -321,6 +333,18 @@ ...@@ -321,6 +333,18 @@
"repositories_verification_total_count": { "type": ["integer", "null"] }, "repositories_verification_total_count": { "type": ["integer", "null"] },
"repositories_verified_in_percentage": { "type": "string" }, "repositories_verified_in_percentage": { "type": "string" },
"repositories_checksum_mismatch_count": { "type": ["integer", "null"] }, "repositories_checksum_mismatch_count": { "type": ["integer", "null"] },
"uploads_count": { "type": ["integer", "null"] },
"uploads_synced_count": { "type": ["integer", "null"] },
"uploads_failed_count": { "type": ["integer", "null"] },
"uploads_registry_count": { "type": ["integer", "null"] },
"uploads_synced_in_percentage": { "type": "string" },
"uploads_checksummed_count": { "type": ["integer", "null"] },
"uploads_checksum_failed_count": { "type": ["integer", "null"] },
"uploads_checksum_total_count": { "type": ["integer", "null"] },
"uploads_verification_failed_count": { "type": ["integer", "null"] },
"uploads_verification_total_count": { "type": ["integer", "null"] },
"uploads_verified_count": { "type": ["integer", "null"] },
"uploads_verified_in_percentage": { "type": "string" },
"wikis_verified_count": { "type": ["integer", "null"] }, "wikis_verified_count": { "type": ["integer", "null"] },
"wikis_verification_failed_count": { "type": ["integer", "null"] }, "wikis_verification_failed_count": { "type": ["integer", "null"] },
"wikis_verification_total_count": { "type": ["integer", "null"] }, "wikis_verification_total_count": { "type": ["integer", "null"] },
......
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Resolvers::Geo::UploadRegistriesResolver do
it_behaves_like 'a Geo registries resolver', :geo_upload_registry
end
...@@ -14,9 +14,8 @@ RSpec.describe GitlabSchema.types['GeoNode'] do ...@@ -14,9 +14,8 @@ RSpec.describe GitlabSchema.types['GeoNode'] do
minimum_reverification_interval merge_request_diff_registries minimum_reverification_interval merge_request_diff_registries
package_file_registries snippet_repository_registries package_file_registries snippet_repository_registries
terraform_state_version_registries group_wiki_repository_registries terraform_state_version_registries group_wiki_repository_registries
lfs_object_registries pages_deployment_registries lfs_object_registries pipeline_artifact_registries
pipeline_artifact_registries upload_registries
pages_deployment_registries
] ]
expect(described_class).to have_graphql_fields(*expected_fields) expect(described_class).to have_graphql_fields(*expected_fields)
......
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe GitlabSchema.types['UploadRegistry'] do
it_behaves_like 'a Geo registry type'
it 'has the expected fields (other than those included in RegistryType)' do
expected_fields = %i[file_id]
expect(described_class).to have_graphql_fields(*expected_fields).at_least
end
end
...@@ -29,7 +29,7 @@ RSpec.describe EE::GeoHelper do ...@@ -29,7 +29,7 @@ RSpec.describe EE::GeoHelper do
repositories repositories
wikis wikis
lfs_objects lfs_objects
attachments uploads
job_artifacts job_artifacts
container_repositories container_repositories
design_repositories design_repositories
......
...@@ -23,7 +23,7 @@ RSpec.describe Gitlab::Geo::GeoNodeStatusCheck do ...@@ -23,7 +23,7 @@ RSpec.describe Gitlab::Geo::GeoNodeStatusCheck do
/Verified Repositories: /, /Verified Repositories: /,
/Wikis: /, /Wikis: /,
/Verified Wikis: /, /Verified Wikis: /,
/Attachments: /, /Uploads: /,
/CI job artifacts: /, /CI job artifacts: /,
/Container repositories: /, /Container repositories: /,
/Design repositories: / /Design repositories: /
......
...@@ -27,7 +27,7 @@ RSpec.describe Gitlab::Geo::LogCursor::Events::UploadDeletedEvent, :clean_gitlab ...@@ -27,7 +27,7 @@ RSpec.describe Gitlab::Geo::LogCursor::Events::UploadDeletedEvent, :clean_gitlab
end end
it 'removes the tracking database entry if exist' do it 'removes the tracking database entry if exist' do
create(:geo_upload_registry, :avatar, file_id: upload.id) create(:geo_upload_legacy_registry, :avatar, file_id: upload.id)
expect { subject.process }.to change(Geo::UploadRegistry, :count).by(-1) expect { subject.process }.to change(Geo::UploadRegistry, :count).by(-1)
end end
......
# frozen_string_literal: true
require 'spec_helper'
require_migration!
RSpec.describe FixStateColumnInFileRegistry do
let(:registry) { table(:file_registry) }
it 'correctly sets registry state value' do
reg0 = registry.create!(file_id: 1, state: 0, success: false, file_type: 'placeholder')
reg1 = registry.create!(file_id: 2, state: 0, success: true, file_type: 'placeholder')
reg2 = registry.create!(file_id: 3, state: 1, success: false, file_type: 'placeholder')
reg3 = registry.create!(file_id: 4, state: 2, success: true, file_type: 'placeholder')
reg4 = registry.create!(file_id: 5, state: 3, success: false, file_type: 'placeholder')
migrate!
expect(registry.where(state: 0)).to contain_exactly(reg0)
expect(registry.where(state: 2)).to contain_exactly(reg1, reg3)
expect(registry.where(state: 1)).to contain_exactly(reg2)
expect(registry.where(state: 3)).to contain_exactly(reg4)
end
end
...@@ -5,13 +5,17 @@ require 'spec_helper' ...@@ -5,13 +5,17 @@ require 'spec_helper'
RSpec.describe Geo::UploadRegistry, :geo do RSpec.describe Geo::UploadRegistry, :geo do
include EE::GeoHelpers include EE::GeoHelpers
before do
stub_feature_flags(geo_upload_replication: false)
end
it_behaves_like 'a BulkInsertSafe model', Geo::UploadRegistry do it_behaves_like 'a BulkInsertSafe model', Geo::UploadRegistry do
let(:valid_items_for_bulk_insertion) { build_list(:geo_upload_registry, 10, created_at: Time.zone.now) } let(:valid_items_for_bulk_insertion) { build_list(:geo_upload_legacy_registry, 10, created_at: Time.zone.now) }
let(:invalid_items_for_bulk_insertion) { [] } # class does not have any validations defined let(:invalid_items_for_bulk_insertion) { [] } # class does not have any validations defined
end end
it 'finds associated Upload record' do it 'finds associated Upload record' do
registry = create(:geo_upload_registry, :attachment, :with_file) registry = create(:geo_upload_legacy_registry, :attachment, :with_file)
expect(described_class.find(registry.id).upload).to be_an_instance_of(Upload) expect(described_class.find(registry.id).upload).to be_an_instance_of(Upload)
end end
...@@ -35,13 +39,13 @@ RSpec.describe Geo::UploadRegistry, :geo do ...@@ -35,13 +39,13 @@ RSpec.describe Geo::UploadRegistry, :geo do
it 'returns untracked IDs as well as tracked IDs that are unused', :aggregate_failures do it 'returns untracked IDs as well as tracked IDs that are unused', :aggregate_failures do
max_id = Upload.maximum(:id) max_id = Upload.maximum(:id)
create(:geo_upload_registry, :avatar, file_id: upload_1.id) create(:geo_upload_legacy_registry, :avatar, file_id: upload_1.id)
create(:geo_upload_registry, :file, file_id: upload_3.id) create(:geo_upload_legacy_registry, :file, file_id: upload_3.id)
create(:geo_upload_registry, :avatar, file_id: upload_5.id) create(:geo_upload_legacy_registry, :avatar, file_id: upload_5.id)
create(:geo_upload_registry, :personal_file, file_id: upload_6.id) create(:geo_upload_legacy_registry, :personal_file, file_id: upload_6.id)
create(:geo_upload_registry, :avatar, file_id: upload_7.id) create(:geo_upload_legacy_registry, :avatar, file_id: upload_7.id)
unused_registry_1 = create(:geo_upload_registry, :attachment, file_id: max_id + 1) unused_registry_1 = create(:geo_upload_legacy_registry, :attachment, file_id: max_id + 1)
unused_registry_2 = create(:geo_upload_registry, :personal_file, file_id: max_id + 2) unused_registry_2 = create(:geo_upload_legacy_registry, :personal_file, file_id: max_id + 2)
range = 1..(max_id + 2) range = 1..(max_id + 2)
untracked, unused = described_class.find_registry_differences(range) untracked, unused = described_class.find_registry_differences(range)
...@@ -65,8 +69,8 @@ RSpec.describe Geo::UploadRegistry, :geo do ...@@ -65,8 +69,8 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.failed' do describe '.failed' do
it 'returns registries in the failed state' do it 'returns registries in the failed state' do
failed = create(:geo_upload_registry, :failed) failed = create(:geo_upload_legacy_registry, :failed)
create(:geo_upload_registry) create(:geo_upload_legacy_registry)
expect(described_class.failed).to match_ids(failed) expect(described_class.failed).to match_ids(failed)
end end
...@@ -74,8 +78,8 @@ RSpec.describe Geo::UploadRegistry, :geo do ...@@ -74,8 +78,8 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.synced' do describe '.synced' do
it 'returns registries in the synced state' do it 'returns registries in the synced state' do
create(:geo_upload_registry, :failed) create(:geo_upload_legacy_registry, :failed)
synced = create(:geo_upload_registry) synced = create(:geo_upload_legacy_registry)
expect(described_class.synced).to match_ids(synced) expect(described_class.synced).to match_ids(synced)
end end
...@@ -83,10 +87,10 @@ RSpec.describe Geo::UploadRegistry, :geo do ...@@ -83,10 +87,10 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.retry_due' do describe '.retry_due' do
it 'returns registries in the synced state' do it 'returns registries in the synced state' do
failed = create(:geo_upload_registry, :failed) failed = create(:geo_upload_legacy_registry, :failed)
synced = create(:geo_upload_registry) synced = create(:geo_upload_legacy_registry)
retry_yesterday = create(:geo_upload_registry, retry_at: Date.yesterday) retry_yesterday = create(:geo_upload_legacy_registry, retry_at: Date.yesterday)
create(:geo_upload_registry, retry_at: Date.tomorrow) create(:geo_upload_legacy_registry, retry_at: Date.tomorrow)
expect(described_class.retry_due).to match_ids([failed, synced, retry_yesterday]) expect(described_class.retry_due).to match_ids([failed, synced, retry_yesterday])
end end
...@@ -94,9 +98,9 @@ RSpec.describe Geo::UploadRegistry, :geo do ...@@ -94,9 +98,9 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.never_attempted_sync' do describe '.never_attempted_sync' do
it 'returns registries that are never synced' do it 'returns registries that are never synced' do
create(:geo_upload_registry, :failed) create(:geo_upload_legacy_registry, :failed)
create(:geo_upload_registry) create(:geo_upload_legacy_registry)
pending = create(:geo_upload_registry, retry_count: nil, success: false) pending = create(:geo_upload_legacy_registry, retry_count: nil, success: false)
expect(described_class.never_attempted_sync).to match_ids([pending]) expect(described_class.never_attempted_sync).to match_ids([pending])
end end
...@@ -125,7 +129,7 @@ RSpec.describe Geo::UploadRegistry, :geo do ...@@ -125,7 +129,7 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '.with_search' do describe '.with_search' do
it 'searches registries on path' do it 'searches registries on path' do
upload = create(:upload, path: 'uploads/-/system/project/avatar/my-awesome-avatar.png') upload = create(:upload, path: 'uploads/-/system/project/avatar/my-awesome-avatar.png')
upload_registry = create(:geo_upload_registry, file_id: upload.id, file_type: :avatar) upload_registry = create(:geo_upload_legacy_registry, file_id: upload.id, file_type: :avatar)
expect(described_class.with_search('awesome-avatar')).to match_ids(upload_registry) expect(described_class.with_search('awesome-avatar')).to match_ids(upload_registry)
end end
...@@ -134,28 +138,28 @@ RSpec.describe Geo::UploadRegistry, :geo do ...@@ -134,28 +138,28 @@ RSpec.describe Geo::UploadRegistry, :geo do
describe '#file' do describe '#file' do
it 'returns the path of the upload of a registry' do it 'returns the path of the upload of a registry' do
upload = create(:upload, :with_file) upload = create(:upload, :with_file)
registry = create(:geo_upload_registry, :file, file_id: upload.id) registry = create(:geo_upload_legacy_registry, :file, file_id: upload.id)
expect(registry.file).to eq(upload.path) expect(registry.file).to eq(upload.path)
end end
it 'return "removed" message when the upload no longer exists' do it 'return "removed" message when the upload no longer exists' do
registry = create(:geo_upload_registry, :avatar) registry = create(:geo_upload_legacy_registry, :avatar)
expect(registry.file).to match(/^Removed avatar with id/) expect(registry.file).to match(/^Removed avatar with id/)
end end
end end
describe '#synchronization_state' do describe '#synchronization_state' do
let_it_be(:failed) { create(:geo_upload_registry, :failed) } let_it_be(:failed) { create(:geo_upload_legacy_registry, :failed) }
let_it_be(:synced) { create(:geo_upload_registry) } let_it_be(:synced) { create(:geo_upload_legacy_registry) }
it 'returns :synced for a successful synced registry' do it 'returns :synced for a successful synced registry' do
expect(synced.synchronization_state).to eq(:synced) expect(synced.synchronization_state).to eq(:synced)
end end
it 'returns :never for a successful registry never synced' do it 'returns :never for a successful registry never synced' do
never = build(:geo_upload_registry, success: false, retry_count: nil) never = build(:geo_upload_legacy_registry, success: false, retry_count: nil)
expect(never.synchronization_state).to eq(:never) expect(never.synchronization_state).to eq(:never)
end end
...@@ -165,3 +169,13 @@ RSpec.describe Geo::UploadRegistry, :geo do ...@@ -165,3 +169,13 @@ RSpec.describe Geo::UploadRegistry, :geo do
end end
end end
end end
RSpec.describe Geo::UploadRegistry, :geo, type: :model do
let_it_be(:registry) { create(:geo_upload_registry) }
specify 'factory is valid' do
expect(registry).to be_valid
end
include_examples 'a Geo framework registry'
end
...@@ -149,39 +149,24 @@ RSpec.describe GeoNodeStatus, :geo do ...@@ -149,39 +149,24 @@ RSpec.describe GeoNodeStatus, :geo do
create_list(:user, 3, avatar: fixture_file_upload('spec/fixtures/dk.png', 'image/png')) create_list(:user, 3, avatar: fixture_file_upload('spec/fixtures/dk.png', 'image/png'))
uploads = Upload.pluck(:id) uploads = Upload.pluck(:id)
create(:geo_upload_registry, :avatar, file_id: uploads[0]) create(:geo_upload_registry, :synced, file_id: uploads[0])
create(:geo_upload_registry, :avatar, file_id: uploads[1]) create(:geo_upload_registry, :synced, file_id: uploads[1])
create(:geo_upload_registry, :avatar, :failed, file_id: uploads[2]) create(:geo_upload_registry, :failed, file_id: uploads[2])
expect(subject.attachments_synced_count).to eq(2) expect(subject.attachments_synced_count).to eq(2)
end end
end end
describe '#attachments_synced_missing_on_primary_count' do
it 'only counts successful syncs' do
create_list(:user, 3, avatar: fixture_file_upload('spec/fixtures/dk.png', 'image/png'))
uploads = Upload.pluck(:id)
create(:geo_upload_registry, :avatar, file_id: uploads[0], missing_on_primary: true)
create(:geo_upload_registry, :avatar, file_id: uploads[1])
create(:geo_upload_registry, :avatar, :failed, file_id: uploads[2])
expect(subject.attachments_synced_missing_on_primary_count).to eq(1)
end
end
describe '#attachments_failed_count' do describe '#attachments_failed_count' do
it 'counts failed avatars, attachment, personal snippets and files' do it 'counts failed avatars, attachment, personal snippets and files' do
# These two should be ignored # These two should be ignored
create(:geo_lfs_object_registry, :failed) create(:geo_lfs_object_registry, :failed)
create(:geo_upload_registry, :with_file) create(:geo_upload_registry)
create(:geo_upload_registry, :with_file, :failed, file_type: :personal_file) create(:geo_upload_registry, :failed)
create(:geo_upload_registry, :with_file, :failed, file_type: :attachment) create(:geo_upload_registry, :failed)
create(:geo_upload_registry, :avatar, :with_file, :failed)
create(:geo_upload_registry, :with_file, :failed)
expect(subject.attachments_failed_count).to eq(4) expect(subject.attachments_failed_count).to eq(2)
end end
end end
...@@ -194,10 +179,10 @@ RSpec.describe GeoNodeStatus, :geo do ...@@ -194,10 +179,10 @@ RSpec.describe GeoNodeStatus, :geo do
create_list(:user, 4, avatar: fixture_file_upload('spec/fixtures/dk.png', 'image/png')) create_list(:user, 4, avatar: fixture_file_upload('spec/fixtures/dk.png', 'image/png'))
uploads = Upload.pluck(:id) uploads = Upload.pluck(:id)
create(:geo_upload_registry, :avatar, file_id: uploads[0]) create(:geo_upload_registry, :synced, file_id: uploads[0])
create(:geo_upload_registry, :avatar, file_id: uploads[1]) create(:geo_upload_registry, :synced, file_id: uploads[1])
create(:geo_upload_registry, :avatar, :failed, file_id: uploads[2]) create(:geo_upload_registry, :failed, file_id: uploads[2])
create(:geo_upload_registry, :avatar, :never_synced, file_id: uploads[3]) create(:geo_upload_registry, :started, file_id: uploads[3])
expect(subject.attachments_synced_in_percentage).to be_within(0.0001).of(50) expect(subject.attachments_synced_in_percentage).to be_within(0.0001).of(50)
end end
...@@ -247,10 +232,7 @@ RSpec.describe GeoNodeStatus, :geo do ...@@ -247,10 +232,7 @@ RSpec.describe GeoNodeStatus, :geo do
it 'counts failed job artifacts' do it 'counts failed job artifacts' do
# These should be ignored # These should be ignored
create(:geo_upload_registry, :failed) create(:geo_upload_registry, :failed)
create(:geo_upload_registry, :avatar, :failed)
create(:geo_upload_registry, :attachment, :failed)
create(:geo_job_artifact_registry, :with_artifact, success: true) create(:geo_job_artifact_registry, :with_artifact, success: true)
create(:geo_job_artifact_registry, :with_artifact, :failed) create(:geo_job_artifact_registry, :with_artifact, :failed)
expect(subject.job_artifacts_failed_count).to eq(1) expect(subject.job_artifacts_failed_count).to eq(1)
...@@ -1130,6 +1112,7 @@ RSpec.describe GeoNodeStatus, :geo do ...@@ -1130,6 +1112,7 @@ RSpec.describe GeoNodeStatus, :geo do
Geo::SnippetRepositoryReplicator | :snippet_repository | :geo_snippet_repository_registry Geo::SnippetRepositoryReplicator | :snippet_repository | :geo_snippet_repository_registry
Geo::GroupWikiRepositoryReplicator | :group_wiki_repository | :geo_group_wiki_repository_registry Geo::GroupWikiRepositoryReplicator | :group_wiki_repository | :geo_group_wiki_repository_registry
Geo::PagesDeploymentReplicator | :pages_deployment | :geo_pages_deployment_registry Geo::PagesDeploymentReplicator | :pages_deployment | :geo_pages_deployment_registry
Geo::UploadReplicator | :upload | :geo_upload_registry
end end
with_them do with_them do
...@@ -1344,8 +1327,8 @@ RSpec.describe GeoNodeStatus, :geo do ...@@ -1344,8 +1327,8 @@ RSpec.describe GeoNodeStatus, :geo do
stub_current_geo_node(primary) stub_current_geo_node(primary)
end end
it 'does not call AttachmentRegistryFinder#registry_count' do it 'does not call AttachmentLegacyRegistryFinder#registry_count' do
expect_any_instance_of(Geo::AttachmentRegistryFinder).not_to receive(:registry_count) expect_any_instance_of(Geo::AttachmentLegacyRegistryFinder).not_to receive(:registry_count)
subject subject
end end
...@@ -1358,8 +1341,8 @@ RSpec.describe GeoNodeStatus, :geo do ...@@ -1358,8 +1341,8 @@ RSpec.describe GeoNodeStatus, :geo do
end end
context 'on the secondary' do context 'on the secondary' do
it 'calls AttachmentRegistryFinder#registry_count' do it 'calls AttachmentLegacyRegistryFinder#registry_count' do
expect_any_instance_of(Geo::AttachmentRegistryFinder).to receive(:registry_count).twice expect_any_instance_of(Geo::AttachmentLegacyRegistryFinder).to receive(:registry_count).twice
subject subject
end end
......
# frozen_string_literal: true
require 'spec_helper'
RSpec.describe Geo::UploadReplicator do
let(:model_record) { create(:upload, :with_file) }
include_examples 'a blob replicator'
end
...@@ -51,4 +51,11 @@ RSpec.describe 'Gets registries' do ...@@ -51,4 +51,11 @@ RSpec.describe 'Gets registries' do
registry_factory: :geo_pages_deployment_registry, registry_factory: :geo_pages_deployment_registry,
registry_foreign_key_field_name: 'pagesDeploymentId' registry_foreign_key_field_name: 'pagesDeploymentId'
} }
it_behaves_like 'gets registries for', {
field_name: 'uploadRegistries',
registry_class_name: 'UploadRegistry',
registry_factory: :geo_upload_registry,
registry_foreign_key_field_name: 'fileId'
}
end end
...@@ -27,14 +27,14 @@ RSpec.describe 'EE-specific admin routing' do ...@@ -27,14 +27,14 @@ RSpec.describe 'EE-specific admin routing' do
end end
describe Admin::Geo::UploadsController, 'routing' do describe Admin::Geo::UploadsController, 'routing' do
let!(:upload_registry) { create(:geo_upload_registry, :with_file, :attachment, success: true) } let!(:upload_registry) { create(:geo_upload_legacy_registry, :with_file, :attachment, success: true) }
it 'routes / to #index' do it 'routes / to #index' do
expect(get('/admin/geo/replication/uploads')).to route_to('admin/geo/uploads#index') expect(get('/admin/geo/replication/legacy-uploads')).to route_to('admin/geo/uploads#index')
end end
it 'routes delete /:id to #destroy' do it 'routes delete /:id to #destroy' do
expect(delete("/admin/geo/replication/uploads/#{upload_registry.id}")).to route_to('admin/geo/uploads#destroy', id: upload_registry.to_param) expect(delete("/admin/geo/replication/legacy-uploads/#{upload_registry.id}")).to route_to('admin/geo/uploads#destroy', id: upload_registry.to_param)
end end
end end
......
...@@ -11,6 +11,8 @@ RSpec.describe Geo::FileDownloadService do ...@@ -11,6 +11,8 @@ RSpec.describe Geo::FileDownloadService do
before do before do
stub_current_geo_node(secondary) stub_current_geo_node(secondary)
stub_feature_flags(geo_upload_replication: false)
end end
describe '#downloader' do describe '#downloader' do
...@@ -58,7 +60,7 @@ RSpec.describe Geo::FileDownloadService do ...@@ -58,7 +60,7 @@ RSpec.describe Geo::FileDownloadService do
context 'with uploads' do context 'with uploads' do
let!(:registry_entry) do let!(:registry_entry) do
create(:geo_upload_registry, :avatar, success: false, file_id: file.id, retry_count: 31) create(:geo_upload_legacy_registry, :avatar, success: false, file_id: file.id, retry_count: 31)
end end
let(:file) { create(:upload) } let(:file) { create(:upload) }
...@@ -233,7 +235,7 @@ RSpec.describe Geo::FileDownloadService do ...@@ -233,7 +235,7 @@ RSpec.describe Geo::FileDownloadService do
when 'job_artifact' when 'job_artifact'
create(:geo_job_artifact_registry, success: false, artifact_id: file.id, retry_count: 3, retry_at: 1.hour.ago) create(:geo_job_artifact_registry, success: false, artifact_id: file.id, retry_count: 3, retry_at: 1.hour.ago)
else else
create(:geo_upload_registry, file_type.to_sym, success: false, file_id: file.id, retry_count: 3, retry_at: 1.hour.ago) create(:geo_upload_legacy_registry, file_type.to_sym, success: false, file_id: file.id, retry_count: 3, retry_at: 1.hour.ago)
end end
end end
......
...@@ -223,7 +223,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do ...@@ -223,7 +223,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
context 'with avatar' do context 'with avatar' do
let!(:upload) { create(:user, :with_avatar).avatar.upload } let!(:upload) { create(:user, :with_avatar).avatar.upload }
let!(:registry) { create(:geo_upload_registry, :avatar, file_id: upload.id) } let!(:registry) { create(:geo_upload_legacy_registry, :avatar, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path } let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes' it_behaves_like 'removes'
...@@ -250,7 +250,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do ...@@ -250,7 +250,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
context 'with attachment' do context 'with attachment' do
let!(:upload) { create(:note, :with_attachment).attachment.upload } let!(:upload) { create(:note, :with_attachment).attachment.upload }
let!(:registry) { create(:geo_upload_registry, :attachment, file_id: upload.id) } let!(:registry) { create(:geo_upload_legacy_registry, :attachment, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path } let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes' it_behaves_like 'removes'
...@@ -284,7 +284,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do ...@@ -284,7 +284,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
Upload.find_by(model: group, uploader: NamespaceFileUploader.name) Upload.find_by(model: group, uploader: NamespaceFileUploader.name)
end end
let!(:registry) { create(:geo_upload_registry, :namespace_file, file_id: upload.id) } let!(:registry) { create(:geo_upload_legacy_registry, :namespace_file, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path } let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes' it_behaves_like 'removes'
...@@ -317,7 +317,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do ...@@ -317,7 +317,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
Upload.find_by(model: snippet, uploader: PersonalFileUploader.name) Upload.find_by(model: snippet, uploader: PersonalFileUploader.name)
end end
let!(:registry) { create(:geo_upload_registry, :personal_file, file_id: upload.id) } let!(:registry) { create(:geo_upload_legacy_registry, :personal_file, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path } let!(:file_path) { upload.retrieve_uploader.file.path }
context 'migrated to object storage' do context 'migrated to object storage' do
...@@ -348,7 +348,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do ...@@ -348,7 +348,7 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
Upload.find_by(model: appearance, uploader: FaviconUploader.name) Upload.find_by(model: appearance, uploader: FaviconUploader.name)
end end
let!(:registry) { create(:geo_upload_registry, :favicon, file_id: upload.id) } let!(:registry) { create(:geo_upload_legacy_registry, :favicon, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path } let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes' it_behaves_like 'removes'
...@@ -406,5 +406,32 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do ...@@ -406,5 +406,32 @@ RSpec.describe Geo::FileRegistryRemovalService, :geo do
end end
end end
end end
context 'with Uploads(after migrating to SSF)' do
let!(:upload) { create(:user, :with_avatar).avatar.upload }
let!(:registry) { create(:geo_upload_registry, file_id: upload.id) }
let!(:file_path) { upload.retrieve_uploader.file.path }
it_behaves_like 'removes'
context 'migrated to object storage' do
before do
stub_uploads_object_storage(AvatarUploader)
upload.update_column(:store, AvatarUploader::Store::REMOTE)
end
context 'with object storage enabled' do
it_behaves_like 'removes'
end
context 'with object storage disabled' do
before do
stub_uploads_object_storage(AvatarUploader, enabled: false)
end
it_behaves_like 'removes registry entry'
end
end
end
end end
end end
...@@ -11,7 +11,7 @@ RSpec.describe Geo::FilesExpireService, :geo do ...@@ -11,7 +11,7 @@ RSpec.describe Geo::FilesExpireService, :geo do
describe '#execute' do describe '#execute' do
let(:file_uploader) { build(:file_uploader, project: project) } let(:file_uploader) { build(:file_uploader, project: project) }
let!(:upload) { Upload.find_by(path: file_uploader.upload_path) } let!(:upload) { Upload.find_by(path: file_uploader.upload_path) }
let!(:upload_registry) { create(:geo_upload_registry, file_id: upload.id) } let!(:upload_registry) { create(:geo_upload_registry, :synced, file_id: upload.id) }
before do before do
project.update(path: "#{project.path}_renamed") project.update(path: "#{project.path}_renamed")
...@@ -32,7 +32,7 @@ RSpec.describe Geo::FilesExpireService, :geo do ...@@ -32,7 +32,7 @@ RSpec.describe Geo::FilesExpireService, :geo do
end end
it 'removes upload_registry associates with upload' do it 'removes upload_registry associates with upload' do
expect(upload_registry.success).to be_truthy expect(upload_registry.synced?).to be_truthy
subject.execute subject.execute
......
...@@ -19,7 +19,8 @@ RSpec.describe Geo::RegistryConsistencyService, :geo, :use_clean_rails_memory_st ...@@ -19,7 +19,8 @@ RSpec.describe Geo::RegistryConsistencyService, :geo, :use_clean_rails_memory_st
{ {
Geo::DesignRegistry => :project_with_design, Geo::DesignRegistry => :project_with_design,
Geo::MergeRequestDiffRegistry => :external_merge_request_diff, Geo::MergeRequestDiffRegistry => :external_merge_request_diff,
Geo::PackageFileRegistry => :package_file Geo::PackageFileRegistry => :package_file,
Geo::UploadRegistry => :upload
}.fetch(registry_class, default_factory_name) }.fetch(registry_class, default_factory_name)
end end
......
...@@ -7,13 +7,13 @@ module EE ...@@ -7,13 +7,13 @@ module EE
override :reset_column_information override :reset_column_information
def reset_column_information(klass) def reset_column_information(klass)
super super
rescue Geo::TrackingBase::SecondaryNotConfigured rescue ::Geo::TrackingBase::SecondaryNotConfigured
end end
override :active_record_base override :active_record_base
def active_record_base def active_record_base
if geo_migration? if geo_migration?
Geo::TrackingBase ::Geo::TrackingBase
else else
super super
end end
......
...@@ -25,6 +25,6 @@ RSpec.shared_examples 'allowlisted /admin/geo requests' do ...@@ -25,6 +25,6 @@ RSpec.shared_examples 'allowlisted /admin/geo requests' do
it_behaves_like 'allowlisted request', :post, '/admin/geo/replication/projects/1/force_redownload' it_behaves_like 'allowlisted request', :post, '/admin/geo/replication/projects/1/force_redownload'
it_behaves_like 'allowlisted request', :delete, '/admin/geo/replication/uploads/1' it_behaves_like 'allowlisted request', :delete, '/admin/geo/replication/legacy-uploads/1'
end end
end end
...@@ -67,6 +67,8 @@ RSpec.shared_examples 'a blob replicator' do ...@@ -67,6 +67,8 @@ RSpec.shared_examples 'a blob replicator' do
describe '#handle_after_destroy' do describe '#handle_after_destroy' do
it 'creates a Geo::Event' do it 'creates a Geo::Event' do
model_record
expect do expect do
replicator.handle_after_destroy replicator.handle_after_destroy
end.to change { ::Geo::Event.count }.by(1) end.to change { ::Geo::Event.count }.by(1)
...@@ -181,7 +183,7 @@ RSpec.shared_examples 'a blob replicator' do ...@@ -181,7 +183,7 @@ RSpec.shared_examples 'a blob replicator' do
context 'when the file is locally stored' do context 'when the file is locally stored' do
context 'when the file exists' do context 'when the file exists' do
it 'returns hexdigest of the file' do it 'returns hexdigest of the file' do
expected = described_class.model.hexdigest(subject.carrierwave_uploader.path) expected = described_class.model.hexdigest(subject.blob_path)
expect(subject.calculate_checksum).to eq(expected) expect(subject.calculate_checksum).to eq(expected)
end end
...@@ -198,7 +200,8 @@ RSpec.shared_examples 'a blob replicator' do ...@@ -198,7 +200,8 @@ RSpec.shared_examples 'a blob replicator' do
context 'when the file is remotely stored' do context 'when the file is remotely stored' do
it 'raises an error' do it 'raises an error' do
allow(subject.carrierwave_uploader).to receive(:file_storage?).and_return(false) carrierwave_uploader = double(file_storage?: false)
allow(subject).to receive(:carrierwave_uploader).and_return(carrierwave_uploader)
expect { subject.calculate_checksum }.to raise_error('File is not checksummable') expect { subject.calculate_checksum }.to raise_error('File is not checksummable')
end end
......
...@@ -330,7 +330,7 @@ RSpec.describe 'geo rake tasks', :geo, :silence_stdout do ...@@ -330,7 +330,7 @@ RSpec.describe 'geo rake tasks', :geo, :silence_stdout do
/Verified Repositories: /, /Verified Repositories: /,
/Wikis: /, /Wikis: /,
/Verified Wikis: /, /Verified Wikis: /,
/Attachments: /, /Uploads: /,
/CI job artifacts: /, /CI job artifacts: /,
/Container repositories: /, /Container repositories: /,
/Design repositories: /, /Design repositories: /,
......
...@@ -17,6 +17,8 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -17,6 +17,8 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end end
WebMock.stub_request(:get, /primary-geo-node/).to_return(status: 200, body: "", headers: {}) WebMock.stub_request(:get, /primary-geo-node/).to_return(status: 200, body: "", headers: {})
stub_feature_flags(geo_upload_replication: false)
end end
it 'does not schedule anything when tracking database is not configured' do it 'does not schedule anything when tracking database is not configured' do
...@@ -48,7 +50,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -48,7 +50,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
let(:upload) { create(:upload) } let(:upload) { create(:upload) }
it 'performs Geo::FileDownloadWorker for unsynced attachments' do it 'performs Geo::FileDownloadWorker for unsynced attachments' do
create(:geo_upload_registry, :avatar, :never_synced, file_id: upload.id) create(:geo_upload_legacy_registry, :avatar, :never_synced, file_id: upload.id)
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('avatar', upload.id) expect(Geo::FileDownloadWorker).to receive(:perform_async).with('avatar', upload.id)
...@@ -56,7 +58,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -56,7 +58,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end end
it 'performs Geo::FileDownloadWorker for failed-sync attachments' do it 'performs Geo::FileDownloadWorker for failed-sync attachments' do
create(:geo_upload_registry, :avatar, :failed, file_id: upload.id, bytes: 0) create(:geo_upload_legacy_registry, :avatar, :failed, file_id: upload.id, bytes: 0)
expect(Geo::FileDownloadWorker).to receive(:perform_async) expect(Geo::FileDownloadWorker).to receive(:perform_async)
.with('avatar', upload.id).once.and_return(spy) .with('avatar', upload.id).once.and_return(spy)
...@@ -65,7 +67,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -65,7 +67,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end end
it 'does not perform Geo::FileDownloadWorker for synced attachments' do it 'does not perform Geo::FileDownloadWorker for synced attachments' do
create(:geo_upload_registry, :avatar, file_id: upload.id, bytes: 1234) create(:geo_upload_legacy_registry, :avatar, file_id: upload.id, bytes: 1234)
expect(Geo::FileDownloadWorker).not_to receive(:perform_async) expect(Geo::FileDownloadWorker).not_to receive(:perform_async)
...@@ -73,7 +75,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -73,7 +75,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end end
it 'does not perform Geo::FileDownloadWorker for synced attachments even with 0 bytes downloaded' do it 'does not perform Geo::FileDownloadWorker for synced attachments even with 0 bytes downloaded' do
create(:geo_upload_registry, :avatar, file_id: upload.id, bytes: 0) create(:geo_upload_legacy_registry, :avatar, file_id: upload.id, bytes: 0)
expect(Geo::FileDownloadWorker).not_to receive(:perform_async) expect(Geo::FileDownloadWorker).not_to receive(:perform_async)
...@@ -81,10 +83,10 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -81,10 +83,10 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end end
context 'with a failed file' do context 'with a failed file' do
let(:failed_registry) { create(:geo_upload_registry, :avatar, :failed, file_id: non_existing_record_id) } let(:failed_registry) { create(:geo_upload_legacy_registry, :avatar, :failed, file_id: non_existing_record_id) }
it 'does not stall backfill' do it 'does not stall backfill' do
unsynced_registry = create(:geo_upload_registry, :avatar, :with_file, :never_synced) unsynced_registry = create(:geo_upload_legacy_registry, :avatar, :with_file, :never_synced)
stub_const('Geo::Scheduler::SchedulerWorker::DB_RETRIEVE_BATCH_SIZE', 1) stub_const('Geo::Scheduler::SchedulerWorker::DB_RETRIEVE_BATCH_SIZE', 1)
...@@ -101,7 +103,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -101,7 +103,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end end
it 'does not retry failed files when retry_at is tomorrow' do it 'does not retry failed files when retry_at is tomorrow' do
failed_registry = create(:geo_upload_registry, :avatar, :failed, file_id: non_existing_record_id, retry_at: Date.tomorrow) failed_registry = create(:geo_upload_legacy_registry, :avatar, :failed, file_id: non_existing_record_id, retry_at: Date.tomorrow)
expect(Geo::FileDownloadWorker).not_to receive(:perform_async).with('avatar', failed_registry.file_id) expect(Geo::FileDownloadWorker).not_to receive(:perform_async).with('avatar', failed_registry.file_id)
...@@ -109,7 +111,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -109,7 +111,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end end
it 'retries failed files when retry_at is in the past' do it 'retries failed files when retry_at is in the past' do
failed_registry = create(:geo_upload_registry, :avatar, :failed, file_id: non_existing_record_id, retry_at: Date.yesterday) failed_registry = create(:geo_upload_legacy_registry, :avatar, :failed, file_id: non_existing_record_id, retry_at: Date.yesterday)
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('avatar', failed_registry.file_id) expect(Geo::FileDownloadWorker).to receive(:perform_async).with('avatar', failed_registry.file_id)
...@@ -131,7 +133,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -131,7 +133,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end end
it 'does not retry those files if there is no spare capacity' do it 'does not retry those files if there is no spare capacity' do
unsynced_registry = create(:geo_upload_registry, :avatar, :with_file, :never_synced) unsynced_registry = create(:geo_upload_legacy_registry, :avatar, :with_file, :never_synced)
expect(subject).to receive(:db_retrieve_batch_size).and_return(1).twice expect(subject).to receive(:db_retrieve_batch_size).and_return(1).twice
expect(Geo::FileDownloadWorker).to receive(:perform_async).with('avatar', unsynced_registry.file_id) expect(Geo::FileDownloadWorker).to receive(:perform_async).with('avatar', unsynced_registry.file_id)
...@@ -140,7 +142,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -140,7 +142,7 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
end end
it 'does not retry those files if they are already scheduled' do it 'does not retry those files if they are already scheduled' do
unsynced_registry = create(:geo_upload_registry, :avatar, :with_file, :never_synced) unsynced_registry = create(:geo_upload_legacy_registry, :avatar, :with_file, :never_synced)
scheduled_jobs = [{ type: 'avatar', id: synced_upload_with_file_missing_on_primary.id, job_id: 'foo' }] scheduled_jobs = [{ type: 'avatar', id: synced_upload_with_file_missing_on_primary.id, job_id: 'foo' }]
expect(subject).to receive(:scheduled_jobs).and_return(scheduled_jobs).at_least(1) expect(subject).to receive(:scheduled_jobs).and_return(scheduled_jobs).at_least(1)
...@@ -276,11 +278,11 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t ...@@ -276,11 +278,11 @@ RSpec.describe Geo::FileDownloadDispatchWorker, :geo, :use_sql_query_cache_for_t
result_object = double(:result, success: true, bytes_downloaded: 100, primary_missing_file: false) result_object = double(:result, success: true, bytes_downloaded: 100, primary_missing_file: false)
allow_any_instance_of(::Gitlab::Geo::Replication::BaseTransfer).to receive(:download_from_primary).and_return(result_object) allow_any_instance_of(::Gitlab::Geo::Replication::BaseTransfer).to receive(:download_from_primary).and_return(result_object)
create_list(:geo_upload_registry, 2, :avatar, :with_file, :never_synced) create_list(:geo_upload_legacy_registry, 2, :avatar, :with_file, :never_synced)
create_list(:geo_upload_registry, 2, :attachment, :with_file, :never_synced) create_list(:geo_upload_legacy_registry, 2, :attachment, :with_file, :never_synced)
create(:geo_upload_registry, :favicon, :with_file, :never_synced) create(:geo_upload_legacy_registry, :favicon, :with_file, :never_synced)
create(:geo_upload_registry, :import_export, :with_file, :never_synced) create(:geo_upload_legacy_registry, :import_export, :with_file, :never_synced)
create(:geo_upload_registry, :personal_file, :with_file, :never_synced) create(:geo_upload_legacy_registry, :personal_file, :with_file, :never_synced)
create(:geo_job_artifact_registry, :with_artifact, :never_synced) create(:geo_job_artifact_registry, :with_artifact, :never_synced)
expect(Geo::FileDownloadWorker).to receive(:perform_async).exactly(8).times.and_call_original expect(Geo::FileDownloadWorker).to receive(:perform_async).exactly(8).times.and_call_original
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment