Merge branch 'mk/update-replicate-a-new-blob-type' into 'master'

Geo: Update instructions to implement replication of a Model

See merge request gitlab-org/gitlab!59343

.gitlab/issue_templates/Geo: Replicate a new Git repository type.md

+<!--
+
+This template is based on a model named `CoolWidget`. 
+
+To adapt this template, find and replace the following tokens:
+
+- `CoolWidget`
+- `Cool Widget`
+- `cool_widget`
+- `coolWidget`
+
+If your Model's pluralized form is non-standard, i.e. it doesn't just end in `s`, then find and replace the following tokens *first*:
+
+- `CoolWidgets`
+- `Cool Widgets`
+- `cool_widgets`
+- `coolWidgets`
+
+-->
+
+## Replicate Cool Widgets
+
+This issue is for implementing Geo replication and verification of Cool Widgets.
+
+For more background, see [Geo self-service framework](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo/framework.md).
+
+In order to implement and test this feature, you need to first [set up Geo locally](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/geo.md).
+
+There are three main sections below. It is a good idea to structure your merge requests this way as well:
+
+1. Modify database schemas to prepare to add Geo support for Cool Widgets
+1. Implement Geo support of Cool Widgets behind a feature flag
+1. Release Geo support of Cool Widgets
+
+It is also a good idea to first open a proof-of-concept merge request. It can be helpful for working out kinks and getting initial support and feedback from the Geo team. As an example, see the [Proof of Concept to replicate Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56423).
+
+### Modify database schemas to prepare to add Geo support for Cool Widgets
+
+You might do this section in its own merge request, but it is not required.
+
+#### Add the registry table to track replication and verification state
+
+Geo secondary sites have a [Geo tracking database](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo.md#tracking-database) independent of the main database. It is used to track the replication and verification state of all replicables. Every Model has a corresponding "registry" table in the Geo tracking database.
+
+- [ ] Create the migration file in `ee/db/geo/migrate`:
+
+  ```shell
+  bin/rails generate geo_migration CreateCoolWidgetRegistry
+  ```
+
+- [ ] Replace the contents of the migration file with the following. Note that we cannot add a foreign key constraint on `cool_widget_id` because the `cool_widgets` table is in a different database. The application code must handle logic such as propagating deletions.
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class CreateCoolWidgetRegistry < ActiveRecord::Migration[6.0]
+    include Gitlab::Database::MigrationHelpers
+
+    disable_ddl_transaction!
+
+    def up
+      unless table_exists?(:cool_widget_registry)
+        ActiveRecord::Base.transaction do
+          create_table :cool_widget_registry, id: :bigserial, force: :cascade do |t|
+            t.bigint :cool_widget_id, null: false
+            t.datetime_with_timezone :created_at, null: false
+            t.datetime_with_timezone :last_synced_at
+            t.datetime_with_timezone :retry_at
+            t.datetime_with_timezone :verified_at
+            t.datetime_with_timezone :verification_started_at
+            t.datetime_with_timezone :verification_retry_at
+            t.integer :state, default: 0, null: false, limit: 2
+            t.integer :verification_state, default: 0, null: false, limit: 2
+            t.integer :retry_count, default: 0, limit: 2, null: false
+            t.integer :verification_retry_count, default: 0, limit: 2, null: false
+            t.boolean :checksum_mismatch, default: false, null: false
+            t.boolean :force_to_redownload, default: false, null: false
+            t.boolean :missing_on_primary, default: false, null: false
+            t.binary :verification_checksum
+            t.binary :verification_checksum_mismatched
+            t.string :verification_failure, limit: 255 # rubocop:disable Migration/PreventStrings see https://gitlab.com/gitlab-org/gitlab/-/issues/323806
+            t.string :last_sync_failure, limit: 255 # rubocop:disable Migration/PreventStrings see https://gitlab.com/gitlab-org/gitlab/-/issues/323806
+
+            t.index :cool_widget_id, name: :index_cool_widget_registry_on_cool_widget_id, unique: true
+            t.index :retry_at
+            t.index :state
+            # To optimize performance of CoolWidgetRegistry.verification_failed_batch
+            t.index :verification_retry_at, name:  :cool_widget_registry_failed_verification, order: "NULLS FIRST",  where: "((state = 2) AND (verification_state = 3))"
+            # To optimize performance of CoolWidgetRegistry.needs_verification_count
+            t.index :verification_state, name:  :cool_widget_registry_needs_verification, where: "((state = 2)  AND (verification_state = ANY (ARRAY[0, 3])))"
+            # To optimize performance of CoolWidgetRegistry.verification_pending_batch
+            t.index :verified_at, name: :cool_widget_registry_pending_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 0))"
+          end
+        end
+      end
+    end
+
+    def down
+      drop_table :cool_widget_registry
+    end
+  end
+  ```
+
+- [ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
+- [ ] Run Geo tracking database migrations:
+
+  ```shell
+  bin/rake geo:db:migrate
+  ```
+
+- [ ] Be sure to commit the relevant changes in `ee/db/geo/schema.rb`
+
+### Add verification state fields on the Geo primary site
+
+The Geo primary site needs to checksum every replicable in order for secondaries to verify their own checksums. To do this, Geo requires fields on the Model. There are two ways to add the necessary verification state fields. If the table is large and wide, then it may be a good idea to add verification state fields to a separate table (Option 2). Consult a database expert if needed.
+
+#### Add verification state fields to the model table (Option 1)
+
+- [ ] Create the migration file in `db/migrate`:
+
+  ```shell
+  bin/rails generate migration AddVerificationStateToCoolWidgets
+  ```
+
+- [ ] Replace the contents of the migration file with:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class AddVerificationStateToCoolWidgets < ActiveRecord::Migration[6.0]
+    def change
+      change_table(:cool_widgets) do |t|
+        t.integer :verification_state, default: 0, limit: 2, null: false
+        t.column :verification_started_at, :datetime_with_timezone
+        t.integer :verification_retry_count, limit: 2, null: false
+        t.column :verification_retry_at, :datetime_with_timezone
+        t.column :verified_at, :datetime_with_timezone
+        t.binary :verification_checksum, using: 'verification_checksum::bytea'
+
+        t.text :verification_failure # rubocop:disable Migration/AddLimitToTextColumns
+      end
+    end
+  end
+  ```
+
+- [ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
+- [ ] If `cool_widgets` is a high-traffic table, follow [the database documentation to use `with_lock_retries`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/migration_style_guide.md#when-to-use-the-helper-method)
+- [ ] Adding a `text` column also [requires](../database/strings_and_the_text_data_type.md#add-a-text-column-to-an-existing-table) setting a limit. Create the migration file in `db/migrate`:
+
+  ```shell
+  bin/rails generate migration AddVerificationFailureLimitToCoolWidgets
+  ```
+
+- [ ] Replace the contents of the migration file with:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class AddVerificationFailureLimitToCoolWidgets < ActiveRecord::Migration[6.0]
+    include Gitlab::Database::MigrationHelpers
+
+    disable_ddl_transaction!
+
+    CONSTRAINT_NAME = 'cool_widget_verification_failure_text_limit'
+
+    def up
+      add_text_limit :cool_widget, :verification_failure, 255, constraint_name: CONSTRAINT_NAME
+    end
+
+    def down
+      remove_check_constraint(:cool_widget, CONSTRAINT_NAME)
+    end
+  end
+  ```
+
+- [ ] Add indexes on verification fields to ensure verification can be performed efficiently. Some or all of these indexes can be omitted if the table is guaranteed to be small. Ask a database expert if you are considering omitting indexes. Create the migration file in `db/migrate`:
+
+  ```shell
+  bin/rails generate migration AddVerificationIndexesToCoolWidgets
+  ```
+
+- [ ] Replace the contents of the migration file with:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class AddVerificationIndexesToCoolWidgets < ActiveRecord::Migration[6.0]
+    include Gitlab::Database::MigrationHelpers
+
+    VERIFICATION_STATE_INDEX_NAME = "index_cool_widgets_on_verification_state"
+    PENDING_VERIFICATION_INDEX_NAME = "index_cool_widgets_pending_verification"
+    FAILED_VERIFICATION_INDEX_NAME = "index_cool_widgets_failed_verification"
+    NEEDS_VERIFICATION_INDEX_NAME = "index_cool_widgets_needs_verification"
+
+    disable_ddl_transaction!
+
+    def up
+      add_concurrent_index :cool_widgets, :verification_state, name: VERIFICATION_STATE_INDEX_NAME
+      add_concurrent_index :cool_widgets, :verified_at, where: "(verification_state = 0)", order: { verified_at: 'ASC NULLS FIRST' }, name: PENDING_VERIFICATION_INDEX_NAME
+      add_concurrent_index :cool_widgets, :verification_retry_at, where: "(verification_state = 3)", order: { verification_retry_at: 'ASC NULLS FIRST' }, name: FAILED_VERIFICATION_INDEX_NAME
+      add_concurrent_index :cool_widgets, :verification_state, where: "(verification_state = 0 OR verification_state = 3)", name: NEEDS_VERIFICATION_INDEX_NAME
+    end
+
+    def down
+      remove_concurrent_index_by_name :cool_widgets, VERIFICATION_STATE_INDEX_NAME
+      remove_concurrent_index_by_name :cool_widgets, PENDING_VERIFICATION_INDEX_NAME
+      remove_concurrent_index_by_name :cool_widgets, FAILED_VERIFICATION_INDEX_NAME
+      remove_concurrent_index_by_name :cool_widgets, NEEDS_VERIFICATION_INDEX_NAME
+    end
+  end
+  ```
+
+- [ ] Run database migrations:
+
+  ```shell
+  bin/rake db:migrate
+  ```
+
+- [ ] Be sure to commit the relevant changes in `db/structure.sql`
+
+#### Add verification state fields to a separate table (Option 2)
+
+- [ ] Create the migration file in `db/migrate`:
+
+  ```shell
+  bin/rails generate migration CreateCoolWidgetStates
+  ```
+
+- [ ] Replace the contents of the migration file with:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class CreateCoolWidgetStates < ActiveRecord::Migration[6.0]
+    include Gitlab::Database::MigrationHelpers
+
+    VERIFICATION_STATE_INDEX_NAME = "index_cool_widget_states_on_verification_state"
+    PENDING_VERIFICATION_INDEX_NAME = "index_cool_widget_states_pending_verification"
+    FAILED_VERIFICATION_INDEX_NAME = "index_cool_widget_states_failed_verification"
+    NEEDS_VERIFICATION_INDEX_NAME = "index_cool_widget_states_needs_verification"
+
+    disable_ddl_transaction!
+
+    def up
+      unless table_exists?(:cool_widget_states)
+        with_lock_retries do
+          create_table :cool_widget_states, id: false do |t|
+            t.references :cool_widget, primary_key: true, null: false, foreign_key: { on_delete: :cascade }
+            t.integer :verification_state, default: 0, limit: 2, null: false
+            t.column :verification_started_at, :datetime_with_timezone
+            t.datetime_with_timezone :verification_retry_at
+            t.datetime_with_timezone :verified_at
+            t.integer :verification_retry_count, limit: 2
+            t.binary :verification_checksum, using: 'verification_checksum::bytea'
+            t.text :verification_failure
+
+            t.index :verification_state, name: VERIFICATION_STATE_INDEX_NAME
+            t.index :verified_at, where: "(verification_state = 0)", order: { verified_at: 'ASC NULLS FIRST' }, name: PENDING_VERIFICATION_INDEX_NAME
+            t.index :verification_retry_at, where: "(verification_state = 3)", order: { verification_retry_at: 'ASC NULLS FIRST' }, name: FAILED_VERIFICATION_INDEX_NAME
+            t.index :verification_state, where: "(verification_state = 0 OR verification_state = 3)", name: NEEDS_VERIFICATION_INDEX_NAME
+          end
+        end
+      end
+
+      add_text_limit :cool_widget_states, :verification_failure, 255
+    end
+
+    def down
+      drop_table :cool_widget_states
+    end
+  end
+  ```
+
+- [ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
+- [ ] Run database migrations:
+
+  ```shell
+  bin/rake db:migrate
+  ```
+
+- [ ] Be sure to commit the relevant changes in `db/structure.sql`
+
+That's all of the required database changes.
+
+### Implement Geo support of Cool Widgets behind a feature flag
+
+#### Step 1. Implement replication and verification
+
+- [ ] Include `Gitlab::Geo::ReplicableModel` in the `CoolWidget` class, and specify the Replicator class `with_replicator Geo::CoolWidgetReplicator`.
+
+  Pay some attention to method `pool_repository`. Not every repository type uses repository pooling. As Geo prefers to use repository snapshotting, it can lead to data loss. Make sure to overwrite `pool_repository` so it returns nil for repositories that do not have pools.
+
+  At this point the `CoolWidget` class should look like this:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class CoolWidget < ApplicationRecord
+    include ::Gitlab::Geo::ReplicableModel
+    include ::Gitlab::Geo::VerificationState
+
+    with_replicator Geo::CoolWidgetReplicator
+
+    mount_uploader :file, CoolWidgetUploader
+
+    # Override the `all` default if not all records can be replicated. For an
+    # example of an existing Model that needs to do this, see
+    # `EE::MergeRequestDiff`.
+    # scope :available_replicables, -> { all }
+
+    # @param primary_key_in [Range, CoolWidget] arg to pass to primary_key_in scope
+    # @return [ActiveRecord::Relation<CoolWidget>] everything that should be synced to this node, restricted by primary key
+    def self.replicables_for_current_secondary(primary_key_in)
+      # This issue template does not help you write this method.
+      #
+      # This method is called only on Geo secondary sites. It is called when
+      # we want to know which records to replicate. This is not easy to automate
+      # because for example:
+      #
+      # * The "selective sync" feature allows admins to choose which namespaces #   to replicate, per secondary site. Most Models are scoped to a
+      #   namespace, but the nature of the relationship to a namespace varies
+      #   between Models.
+      # * The "selective sync" feature allows admins to choose which shards to
+      #   replicate, per secondary site. Repositories are associated with
+      #   shards. Most blob types are not, but Project Uploads are.
+      # * Remote stored replicables are not replicated, by default. But the
+      #   setting `sync_object_storage` enables replication of remote stored
+      #   replicables.
+      #
+      # Search the codebase for examples, and consult a Geo expert if needed.
+    end
+
+    # Geo checks this method in FrameworkRepositorySyncService to avoid
+    # snapshotting repositories using object pools
+    def pool_repository
+      nil
+    end
+
+    ...
+  end
+  ```
+
+- [ ] Implement `CoolWidget.replicables_for_current_secondary` above.
+- [ ] Ensure `CoolWidget.replicables_for_current_secondary` is well-tested. Search the codebase for `replicables_for_current_secondary` to find examples of parameterized table specs. You may need to add more `FactoryBot` traits.
+- [ ] If you are using a separate table `cool_widget_states` to track verification state on the Geo primary site, then:
+  - [ ] Do not include `::Gitlab::Geo::VerificationState` on the `CoolWidget` class.
+  - [ ] Add the following lines to the `cool_widget_state.rb` model:
+
+    ```ruby
+    class CoolWidgetState < ApplicationRecord
+      ...
+      self.primary_key = :cool_widget_id
+
+      include ::Gitlab::Geo::VerificationState
+
+      belongs_to :cool_widget, inverse_of: :cool_widget_state
+      ...
+    end
+    ```
+
+  - [ ] Add the following lines to the `cool_widget` model:
+
+    ```ruby
+    class CoolWidget < ApplicationRecord
+      ...
+      has_one :cool_widget_state, inverse_of: :cool_widget
+
+      delegate :verification_retry_at, :verification_retry_at=,
+              :verified_at, :verified_at=,
+              :verification_checksum, :verification_checksum=,
+              :verification_failure, :verification_failure=,
+              :verification_retry_count, :verification_retry_count=,
+              to: :cool_widget_state
+      ...
+    end
+    ```
+
+- [ ] Create `ee/app/replicators/geo/cool_widget_replicator.rb`. Implement the `#repository` method which should return a `<Repository>` instance, and implement the class method `.model` to return the `CoolWidget` class:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  module Geo
+    class CoolWidgetReplicator < Gitlab::Geo::Replicator
+      include ::Geo::RepositoryReplicatorStrategy
+
+      def self.model
+        ::CoolWidget
+      end
+
+      def repository
+        model_record.repository
+      end
+
+      def self.git_access_class
+        ::Gitlab::GitAccessCoolWidget
+      end
+
+      # The feature flag follows the format `geo_#{replicable_name}_replication`,
+      # so here it would be `geo_cool_widget_replication`
+      def self.replication_enabled_by_default?
+        false
+      end
+
+      override :verification_feature_flag_enabled?
+      def self.verification_feature_flag_enabled?
+        # We are adding verification at the same time as replication, so we
+        # don't need to toggle verification separately from replication. When
+        # the replication feature flag is off, then verification is also off
+        # (see `VerifiableReplicator.verification_enabled?`)
+        true
+      end
+
+    end
+  end
+  ```
+
+- [ ] Make sure Geo push events are created. Usually it needs some change in the `app/workers/post_receive.rb` file. Example:
+
+  ```ruby
+  def replicate_cool_widget_changes(cool_widget)
+    if ::Gitlab::Geo.primary?
+      cool_widget.replicator.handle_after_update if cool_widget
+    end
+  end
+  ```
+
+  See `app/workers/post_receive.rb` for more examples.
+
+- [ ] Make sure the repository removal is also handled. You may need to add something like the following in the destroy service of the repository:
+
+  ```ruby
+  cool_widget.replicator.handle_after_destroy if cool_widget.repository
+  ```
+
+- [ ] Make sure a Geo secondary site can request and download Cool Widgets on the Geo primary site. You may need to make some changes to `Gitlab::GitAccessCoolWidget`. For example, see [this change for Group-level Wikis](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54914/diffs?commit_id=0f2b36f66697b4addbc69bd377ee2818f648dd33).
+- [ ] Generate the feature flag definition file by running the feature flag command and following the command prompts:
+
+  ```shell
+  bin/feature-flag --ee geo_cool_widget_replication --type development --group 'group::geo'
+  ```
+
+- [ ] Add this replicator class to the method `replicator_classes` in
+  `ee/lib/gitlab/geo.rb`:
+
+  ```ruby
+  REPLICATOR_CLASSES = [
+    ::Geo::PackageFileReplicator,
+    ::Geo::CoolWidgetReplicator
+  ]
+  end
+  ```
+
+- [ ] Create `ee/spec/replicators/geo/cool_widget_replicator_spec.rb` and perform the necessary setup to define the `model_record` variable for the shared examples:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe Geo::CoolWidgetReplicator do
+    let(:model_record) { build(:cool_widget) }
+
+    include_examples 'a repository replicator'
+    include_examples 'a verifiable replicator'
+  end
+  ```
+
+- [ ] Create `ee/app/models/geo/cool_widget_registry.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class Geo::CoolWidgetRegistry < Geo::BaseRegistry
+    include ::Geo::ReplicableRegistry
+    include ::Geo::VerifiableRegistry
+
+    MODEL_CLASS = ::CoolWidget
+    MODEL_FOREIGN_KEY = :cool_widget_id
+
+    belongs_to :cool_widget, class_name: 'CoolWidget'
+  end
+  ```
+
+- [ ] Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`.
+- [ ] Update `def model_class_factory_name` in `ee/spec/services/geo/registry_consistency_service_spec.rb`.
+- [ ] Update `it 'creates missing registries for each registry class'` in `ee/spec/workers/geo/secondary/registry_consistency_worker_spec.rb`.
+- [ ] Add `cool_widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`.
+- [ ] Create `ee/spec/factories/geo/cool_widget_registry.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  FactoryBot.define do
+    factory :geo_cool_widget_registry, class: 'Geo::CoolWidgetRegistry' do
+      cool_widget
+      state { Geo::CoolWidgetRegistry.state_value(:pending) }
+
+      trait :synced do
+        state { Geo::CoolWidgetRegistry.state_value(:synced) }
+        last_synced_at { 5.days.ago }
+      end
+
+      trait :failed do
+        state { Geo::CoolWidgetRegistry.state_value(:failed) }
+        last_synced_at { 1.day.ago }
+        retry_count { 2 }
+        last_sync_failure { 'Random error' }
+      end
+
+      trait :started do
+        state { Geo::CoolWidgetRegistry.state_value(:started) }
+        last_synced_at { 1.day.ago }
+        retry_count { 0 }
+      end
+    end
+  end
+  ```
+
+- [ ] Create `ee/spec/models/geo/cool_widget_registry_spec.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe Geo::CoolWidgetRegistry, :geo, type: :model do
+    let_it_be(:registry) { create(:geo_cool_widget_registry) }
+
+    specify 'factory is valid' do
+      expect(registry).to be_valid
+    end
+
+    include_examples 'a Geo framework registry'
+    include_examples 'a Geo verifiable registry'
+  end
+  ```
+
+#### Step 2. Implement metrics gathering
+
+Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in `GeoNodeStatus` for display in the UI, and sent to Prometheus:
+
+- [ ] Add the following fields to Geo Node Status example responses in `doc/api/geo_nodes.md`:
+  - `cool_widgets_count`
+  - `cool_widgets_checksum_total_count`
+  - `cool_widgets_checksummed_count`
+  - `cool_widgets_checksum_failed_count`
+  - `cool_widgets_synced_count`
+  - `cool_widgets_failed_count`
+  - `cool_widgets_registry_count`
+  - `cool_widgets_verification_total_count`
+  - `cool_widgets_verified_count`
+  - `cool_widgets_verification_failed_count`
+  - `cool_widgets_synced_in_percentage`
+  - `cool_widgets_verified_in_percentage`
+- [ ] Add the same fields to `GET /geo_nodes/status` example response in
+  `ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json`.
+- [ ] Add the following fields to the `Sidekiq metrics` table in `doc/administration/monitoring/prometheus/gitlab_metrics.md`:
+  - `geo_cool_widgets`
+  - `geo_cool_widgets_checksum_total`
+  - `geo_cool_widgets_checksummed`
+  - `geo_cool_widgets_checksum_failed`
+  - `geo_cool_widgets_synced`
+  - `geo_cool_widgets_failed`
+  - `geo_cool_widgets_registry`
+  - `geo_cool_widgets_verification_total`
+  - `geo_cool_widgets_verified`
+  - `geo_cool_widgets_verification_failed`
+- [ ] Add the following to the parameterized table in the `context 'Replicator stats' do` block in `ee/spec/models/geo_node_status_spec.rb`:
+
+  ```ruby
+  Geo::CoolWidgetReplicator | :cool_widget | :geo_cool_widget_registry
+  ```
+
+- [ ] Add the following to `spec/factories/cool_widgets.rb`:
+
+  ```ruby
+  trait(:verification_succeeded) do
+    with_file
+    verification_checksum { 'abc' }
+    verification_state { CoolWidget.verification_state_value(:verification_succeeded) }
+  end
+
+  trait(:verification_failed) do
+    with_file
+    verification_failure { 'Could not calculate the checksum' }
+    verification_state { CoolWidget.verification_state_value(:verification_failed) }
+  end
+  ```
+
+- [ ] Make sure the factory also allows setting a `project` attribute. If the model does not have a direct relation to a project, you can use a `transient` attribute. Check out `spec/factories/merge_request_diffs.rb` for an example.
+
+Cool Widget replication and verification metrics should now be available in the API, the `Admin > Geo > Nodes` view, and Prometheus.
+
+#### Step 3. Implement the GraphQL API
+
+The GraphQL API is used by `Admin > Geo > Replication Details` views, and is directly queryable by administrators.
+
+- [ ] Add a new field to `GeoNodeType` in `ee/app/graphql/types/geo/geo_node_type.rb`:
+
+  ```ruby
+  field :cool_widget_registries, ::Types::Geo::CoolWidgetRegistryType.connection_type,
+        null: true,
+        resolver: ::Resolvers::Geo::CoolWidgetRegistriesResolver,
+        description: 'Find Cool Widget registries on this Geo node',
+        feature_flag: :geo_cool_widget_replication
+  ```
+
+- [ ] Add the new `cool_widget_registries` field name to the `expected_fields` array in `ee/spec/graphql/types/geo/geo_node_type_spec.rb`.
+- [ ] Create `ee/app/graphql/resolvers/geo/cool_widget_registries_resolver.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  module Resolvers
+    module Geo
+      class CoolWidgetRegistriesResolver < BaseResolver
+        type ::Types::Geo::GeoNodeType.connection_type, null: true
+
+        include RegistriesResolver
+      end
+    end
+  end
+  ```
+
+- [ ] Create `ee/spec/graphql/resolvers/geo/cool_widget_registries_resolver_spec.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe Resolvers::Geo::CoolWidgetRegistriesResolver do
+    it_behaves_like 'a Geo registries resolver', :geo_cool_widget_registry
+  end
+  ```
+
+- [ ] Create `ee/app/finders/geo/cool_widget_registry_finder.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  module Geo
+    class CoolWidgetRegistryFinder
+      include FrameworkRegistryFinder
+    end
+  end
+  ```
+
+- [ ] Create `ee/spec/finders/geo/cool_widget_registry_finder_spec.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe Geo::CoolWidgetRegistryFinder do
+    it_behaves_like 'a framework registry finder', :geo_cool_widget_registry
+  end
+  ```
+
+- [ ] Create `ee/app/graphql/types/geo/cool_widget_registry_type.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  module Types
+    module Geo
+      # rubocop:disable Graphql/AuthorizeTypes because it is included
+      class CoolWidgetRegistryType < BaseObject
+        include ::Types::Geo::RegistryType
+
+        graphql_name 'CoolWidgetRegistry'
+        description 'Represents the Geo replication and verification state of a cool_widget'
+
+        field :cool_widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Cool Widget'
+      end
+    end
+  end
+  ```
+
+- [ ] Create `ee/spec/graphql/types/geo/cool_widget_registry_type_spec.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe GitlabSchema.types['CoolWidgetRegistry'] do
+    it_behaves_like 'a Geo registry type'
+
+    it 'has the expected fields (other than those included in RegistryType)' do
+      expected_fields = %i[cool_widget_id]
+
+      expect(described_class).to have_graphql_fields(*expected_fields).at_least
+    end
+  end
+  ```
+
+- [ ] Add integration tests for providing CoolWidget registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in `ee/spec/requests/api/graphql/geo/registries_spec.rb`:
+
+  ```ruby
+  it_behaves_like 'gets registries for', {
+    field_name: 'coolWidgetRegistries',
+    registry_class_name: 'CoolWidgetRegistry',
+    registry_factory: :geo_cool_widget_registry,
+    registry_foreign_key_field_name: 'coolWidgetId'
+  }
+  ```
+
+- [ ] Update the GraphQL reference documentation:
+
+  ```shell
+  bundle exec rake gitlab:graphql:compile_docs
+  ```
+
+Individual Cool Widget replication and verification data should now be available via the GraphQL API.
+
+### Release Geo support of Cool Widgets
+
+- [ ] In the rollout issue you created when creating the feature flag, modify the Roll Out Steps:
+  - [ ] Cross out any steps related to testing on production GitLab.com, because Geo is not running on production GitLab.com at the moment.
+  - [ ] Add a step to `Test replication and verification of Cool Widgets on a non-GDK-deployment. For example, using GitLab Environment Toolkit`.
+  - [ ] Add a step to `Ping the Geo PM and EM to coordinate testing`. For example, you might add steps to generate Cool Widgets, and then a Geo engineer may take it from there.
+- [ ] In `ee/config/feature_flags/development/geo_cool_widget_replication.yml`, set `default_enabled: true`
+
+- [ ] In `ee/app/replicators/geo/cool_widget_replicator.rb`, delete the `self.replication_enabled_by_default?` method:
+
+  ```ruby
+  module Geo
+    class CoolWidgetReplicator < Gitlab::Geo::Replicator
+      ...
+
+      # REMOVE THIS METHOD
+      def self.replication_enabled_by_default?
+        false
+      end
+      # REMOVE THIS METHOD
+
+      ...
+    end
+  end
+  ```
+
+- [ ] In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `feature_flag` option for the released type:
+
+  ```ruby
+  field :cool_widget_registries, ::Types::Geo::CoolWidgetRegistryType.connection_type,
+        null: true,
+        resolver: ::Resolvers::Geo::CoolWidgetRegistriesResolver,
+        description: 'Find Cool Widget registries on this Geo node',
+        feature_flag: :geo_cool_widget_replication # REMOVE THIS LINE
+  ```
+
+- [ ] Add a row for Cool Widgets to the `Data types` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#data-types)
+- [ ] Add a row for Cool Widgets to the `Limitations on replication/verification` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#limitations-on-replicationverification). If the row already exists, then update it to show that Replication and Verification is released in the current version.

.gitlab/issue_templates/Geo: Replicate a new blob type.md

+<!--
+
+This template is based on a model named `CoolWidget`. 
+
+To adapt this template, find and replace the following tokens:
+
+- `CoolWidget`
+- `Cool Widget`
+- `cool_widget`
+- `coolWidget`
+
+If your Model's pluralized form is non-standard, i.e. it doesn't just end in `s`, find and replace the following tokens *first*:
+
+- `CoolWidgets`
+- `Cool Widgets`
+- `cool_widgets`
+- `coolWidgets`
+
+-->
+
+## Replicate Cool Widgets
+
+This issue is for implementing Geo replication and verification of Cool Widgets.
+
+For more background, see [Geo self-service framework](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo/framework.md).
+
+In order to implement and test this feature, you need to first [set up Geo locally](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/geo.md).
+
+There are three main sections below. It is a good idea to structure your merge requests this way as well:
+
+1. Modify database schemas to prepare to add Geo support for Cool Widgets
+1. Implement Geo support of Cool Widgets behind a feature flag
+1. Release Geo support of Cool Widgets
+
+It is also a good idea to first open a proof-of-concept merge request. It can be helpful for working out kinks and getting initial support and feedback from the Geo team. As an example, see the [Proof of Concept to replicate Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56423).
+
+### Modify database schemas to prepare to add Geo support for Cool Widgets
+
+You might do this section in its own merge request, but it is not required.
+
+#### Add the registry table to track replication and verification state
+
+Geo secondary sites have a [Geo tracking database](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/geo.md#tracking-database) independent of the main database. It is used to track the replication and verification state of all replicables. Every Model has a corresponding "registry" table in the Geo tracking database.
+
+- [ ] Create the migration file in `ee/db/geo/migrate`:
+
+  ```shell
+  bin/rails generate geo_migration CreateCoolWidgetRegistry
+  ```
+
+- [ ] Replace the contents of the migration file with the following. Note that we cannot add a foreign key constraint on `cool_widget_id` because the `cool_widgets` table is in a different database. The application code must handle logic such as propagating deletions.
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class CreateCoolWidgetRegistry < ActiveRecord::Migration[6.0]
+    include Gitlab::Database::MigrationHelpers
+
+    disable_ddl_transaction!
+
+    def up
+      unless table_exists?(:cool_widget_registry)
+        ActiveRecord::Base.transaction do
+          create_table :cool_widget_registry, id: :bigserial, force: :cascade do |t|
+            t.bigint :cool_widget_id, null: false
+            t.datetime_with_timezone :created_at, null: false
+            t.datetime_with_timezone :last_synced_at
+            t.datetime_with_timezone :retry_at
+            t.datetime_with_timezone :verified_at
+            t.datetime_with_timezone :verification_started_at
+            t.datetime_with_timezone :verification_retry_at
+            t.integer :state, default: 0, null: false, limit: 2
+            t.integer :verification_state, default: 0, null: false, limit: 2
+            t.integer :retry_count, default: 0, limit: 2, null: false
+            t.integer :verification_retry_count, default: 0, limit: 2, null: false
+            t.boolean :checksum_mismatch, default: false, null: false
+            t.binary :verification_checksum
+            t.binary :verification_checksum_mismatched
+            t.string :verification_failure, limit: 255 # rubocop:disable Migration/PreventStrings see https://gitlab.com/gitlab-org/gitlab/-/issues/323806
+            t.string :last_sync_failure, limit: 255 # rubocop:disable Migration/PreventStrings see https://gitlab.com/gitlab-org/gitlab/-/issues/323806
+
+            t.index :cool_widget_id, name: :index_cool_widget_registry_on_cool_widget_id, unique: true
+            t.index :retry_at
+            t.index :state
+            # To optimize performance of CoolWidgetRegistry.verification_failed_batch
+            t.index :verification_retry_at, name:  :cool_widget_registry_failed_verification, order: "NULLS FIRST",  where: "((state = 2) AND (verification_state = 3))"
+            # To optimize performance of CoolWidgetRegistry.needs_verification_count
+            t.index :verification_state, name:  :cool_widget_registry_needs_verification, where: "((state = 2)  AND (verification_state = ANY (ARRAY[0, 3])))"
+            # To optimize performance of CoolWidgetRegistry.verification_pending_batch
+            t.index :verified_at, name: :cool_widget_registry_pending_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 0))"
+          end
+        end
+      end
+    end
+
+    def down
+      drop_table :cool_widget_registry
+    end
+  end
+  ```
+
+- [ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
+- [ ] Run Geo tracking database migrations:
+
+  ```shell
+  bin/rake geo:db:migrate
+  ```
+
+- [ ] Be sure to commit the relevant changes in `ee/db/geo/schema.rb`
+
+### Add verification state fields on the Geo primary site
+
+The Geo primary site needs to checksum every replicable in order for secondaries to verify their own checksums. To do this, Geo requires fields on the Model. There are two ways to add the necessary verification state fields. If the table is large and wide, then it may be a good idea to add verification state fields to a separate table (Option 2). Consult a database expert if needed.
+
+#### Add verification state fields to the model table (Option 1)
+
+- [ ] Create the migration file in `db/migrate`:
+
+  ```shell
+  bin/rails generate migration AddVerificationStateToCoolWidgets
+  ```
+
+- [ ] Replace the contents of the migration file with:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class AddVerificationStateToCoolWidgets < ActiveRecord::Migration[6.0]
+    def change
+      change_table(:cool_widgets) do |t|
+        t.integer :verification_state, default: 0, limit: 2, null: false
+        t.column :verification_started_at, :datetime_with_timezone
+        t.integer :verification_retry_count, limit: 2, null: false
+        t.column :verification_retry_at, :datetime_with_timezone
+        t.column :verified_at, :datetime_with_timezone
+        t.binary :verification_checksum, using: 'verification_checksum::bytea'
+
+        t.text :verification_failure # rubocop:disable Migration/AddLimitToTextColumns
+      end
+    end
+  end
+  ```
+
+- [ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
+- [ ] If `cool_widgets` is a high-traffic table, follow [the database documentation to use `with_lock_retries`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/migration_style_guide.md#when-to-use-the-helper-method)
+- [ ] Adding a `text` column also [requires](../database/strings_and_the_text_data_type.md#add-a-text-column-to-an-existing-table) setting a limit. Create the migration file in `db/migrate`:
+
+  ```shell
+  bin/rails generate migration AddVerificationFailureLimitToCoolWidgets
+  ```
+
+- [ ] Replace the contents of the migration file with:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class AddVerificationFailureLimitToCoolWidgets < ActiveRecord::Migration[6.0]
+    include Gitlab::Database::MigrationHelpers
+
+    disable_ddl_transaction!
+
+    CONSTRAINT_NAME = 'cool_widget_verification_failure_text_limit'
+
+    def up
+      add_text_limit :cool_widget, :verification_failure, 255, constraint_name: CONSTRAINT_NAME
+    end
+
+    def down
+      remove_check_constraint(:cool_widget, CONSTRAINT_NAME)
+    end
+  end
+  ```
+
+- [ ] Add indexes on verification fields to ensure verification can be performed efficiently. Some or all of these indexes can be omitted if the table is guaranteed to be small. Ask a database expert if you are considering omitting indexes. Create the migration file in `db/migrate`:
+
+  ```shell
+  bin/rails generate migration AddVerificationIndexesToCoolWidgets
+  ```
+
+- [ ] Replace the contents of the migration file with:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class AddVerificationIndexesToCoolWidgets < ActiveRecord::Migration[6.0]
+    include Gitlab::Database::MigrationHelpers
+
+    VERIFICATION_STATE_INDEX_NAME = "index_cool_widgets_on_verification_state"
+    PENDING_VERIFICATION_INDEX_NAME = "index_cool_widgets_pending_verification"
+    FAILED_VERIFICATION_INDEX_NAME = "index_cool_widgets_failed_verification"
+    NEEDS_VERIFICATION_INDEX_NAME = "index_cool_widgets_needs_verification"
+
+    disable_ddl_transaction!
+
+    def up
+      add_concurrent_index :cool_widgets, :verification_state, name: VERIFICATION_STATE_INDEX_NAME
+      add_concurrent_index :cool_widgets, :verified_at, where: "(verification_state = 0)", order: { verified_at: 'ASC NULLS FIRST' }, name: PENDING_VERIFICATION_INDEX_NAME
+      add_concurrent_index :cool_widgets, :verification_retry_at, where: "(verification_state = 3)", order: { verification_retry_at: 'ASC NULLS FIRST' }, name: FAILED_VERIFICATION_INDEX_NAME
+      add_concurrent_index :cool_widgets, :verification_state, where: "(verification_state = 0 OR verification_state = 3)", name: NEEDS_VERIFICATION_INDEX_NAME
+    end
+
+    def down
+      remove_concurrent_index_by_name :cool_widgets, VERIFICATION_STATE_INDEX_NAME
+      remove_concurrent_index_by_name :cool_widgets, PENDING_VERIFICATION_INDEX_NAME
+      remove_concurrent_index_by_name :cool_widgets, FAILED_VERIFICATION_INDEX_NAME
+      remove_concurrent_index_by_name :cool_widgets, NEEDS_VERIFICATION_INDEX_NAME
+    end
+  end
+  ```
+
+- [ ] Run database migrations:
+
+  ```shell
+  bin/rake db:migrate
+  ```
+
+- [ ] Be sure to commit the relevant changes in `db/structure.sql`
+
+#### Add verification state fields to a separate table (Option 2)
+
+- [ ] Create the migration file in `db/migrate`:
+
+  ```shell
+  bin/rails generate migration CreateCoolWidgetStates
+  ```
+
+- [ ] Replace the contents of the migration file with:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class CreateCoolWidgetStates < ActiveRecord::Migration[6.0]
+    include Gitlab::Database::MigrationHelpers
+
+    VERIFICATION_STATE_INDEX_NAME = "index_cool_widget_states_on_verification_state"
+    PENDING_VERIFICATION_INDEX_NAME = "index_cool_widget_states_pending_verification"
+    FAILED_VERIFICATION_INDEX_NAME = "index_cool_widget_states_failed_verification"
+    NEEDS_VERIFICATION_INDEX_NAME = "index_cool_widget_states_needs_verification"
+
+    disable_ddl_transaction!
+
+    def up
+      unless table_exists?(:cool_widget_states)
+        with_lock_retries do
+          create_table :cool_widget_states, id: false do |t|
+            t.references :cool_widget, primary_key: true, null: false, foreign_key: { on_delete: :cascade }
+            t.integer :verification_state, default: 0, limit: 2, null: false
+            t.column :verification_started_at, :datetime_with_timezone
+            t.datetime_with_timezone :verification_retry_at
+            t.datetime_with_timezone :verified_at
+            t.integer :verification_retry_count, limit: 2
+            t.binary :verification_checksum, using: 'verification_checksum::bytea'
+            t.text :verification_failure
+
+            t.index :verification_state, name: VERIFICATION_STATE_INDEX_NAME
+            t.index :verified_at, where: "(verification_state = 0)", order: { verified_at: 'ASC NULLS FIRST' }, name: PENDING_VERIFICATION_INDEX_NAME
+            t.index :verification_retry_at, where: "(verification_state = 3)", order: { verification_retry_at: 'ASC NULLS FIRST' }, name: FAILED_VERIFICATION_INDEX_NAME
+            t.index :verification_state, where: "(verification_state = 0 OR verification_state = 3)", name: NEEDS_VERIFICATION_INDEX_NAME
+          end
+        end
+      end
+
+      add_text_limit :cool_widget_states, :verification_failure, 255
+    end
+
+    def down
+      drop_table :cool_widget_states
+    end
+  end
+  ```
+
+- [ ] If deviating from the above example, then be sure to order columns according to [our guidelines](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/development/ordering_table_columns.md).
+- [ ] Run database migrations:
+
+  ```shell
+  bin/rake db:migrate
+  ```
+
+- [ ] Be sure to commit the relevant changes in `db/structure.sql`
+
+That's all of the required database changes.
+
+### Implement Geo support of Cool Widgets behind a feature flag
+
+#### Step 1. Implement replication and verification
+
+- [ ] Include `Gitlab::Geo::ReplicableModel` in the `CoolWidget` class, and specify the Replicator class `with_replicator Geo::CoolWidgetReplicator`.
+
+  At this point the `CoolWidget` class should look like this:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class CoolWidget < ApplicationRecord
+    include ::Gitlab::Geo::ReplicableModel
+    include ::Gitlab::Geo::VerificationState
+
+    with_replicator Geo::CoolWidgetReplicator
+
+    mount_uploader :file, CoolWidgetUploader
+
+    # Override the `all` default if not all records can be replicated. For an
+    # example of an existing Model that needs to do this, see
+    # `EE::MergeRequestDiff`.
+    # scope :available_replicables, -> { all }
+
+    # @param primary_key_in [Range, CoolWidget] arg to pass to primary_key_in scope
+    # @return [ActiveRecord::Relation<CoolWidget>] everything that should be synced to this node, restricted by primary key
+    def self.replicables_for_current_secondary(primary_key_in)
+      # This issue template does not help you write this method.
+      #
+      # This method is called only on Geo secondary sites. It is called when
+      # we want to know which records to replicate. This is not easy to automate
+      # because for example:
+      #
+      # * The "selective sync" feature allows admins to choose which namespaces #   to replicate, per secondary site. Most Models are scoped to a
+      #   namespace, but the nature of the relationship to a namespace varies
+      #   between Models.
+      # * The "selective sync" feature allows admins to choose which shards to
+      #   replicate, per secondary site. Repositories are associated with
+      #   shards. Most blob types are not, but Project Uploads are.
+      # * Remote stored replicables are not replicated, by default. But the
+      #   setting `sync_object_storage` enables replication of remote stored
+      #   replicables.
+      #
+      # Search the codebase for examples, and consult a Geo expert if needed.
+    end
+    ...
+  end
+  ```
+
+- [ ] Implement `CoolWidget.replicables_for_current_secondary` above.
+- [ ] Ensure `CoolWidget.replicables_for_current_secondary` is well-tested. Search the codebase for `replicables_for_current_secondary` to find examples of parameterized table specs. You may need to add more `FactoryBot` traits.
+- [ ] If you are using a separate table `cool_widget_states` to track verification state on the Geo primary site, then:
+  - [ ] Do not include `::Gitlab::Geo::VerificationState` on the `CoolWidget` class.
+  - [ ] Add the following lines to the `cool_widget_state.rb` model:
+
+    ```ruby
+    class CoolWidgetState < ApplicationRecord
+      ...
+      self.primary_key = :cool_widget_id
+
+      include ::Gitlab::Geo::VerificationState
+
+      belongs_to :cool_widget, inverse_of: :cool_widget_state
+      ...
+    end
+    ```
+
+  - [ ] Add the following lines to the `cool_widget` model:
+
+    ```ruby
+    class CoolWidget < ApplicationRecord
+      ...
+      has_one :cool_widget_state, inverse_of: :cool_widget
+
+      delegate :verification_retry_at, :verification_retry_at=,
+              :verified_at, :verified_at=,
+              :verification_checksum, :verification_checksum=,
+              :verification_failure, :verification_failure=,
+              :verification_retry_count, :verification_retry_count=,
+              to: :cool_widget_state
+      ...
+    end
+    ```
+
+- [ ] Create `ee/app/replicators/geo/cool_widget_replicator.rb`. Implement the `#carrierwave_uploader` method which should return a `CarrierWave::Uploader`, and implement the class method `.model` to return the `CoolWidget` class:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  module Geo
+    class CoolWidgetReplicator < Gitlab::Geo::Replicator
+      include ::Geo::BlobReplicatorStrategy
+
+      def self.model
+        ::CoolWidget
+      end
+
+      def carrierwave_uploader
+        model_record.file
+      end
+
+      # The feature flag follows the format `geo_#{replicable_name}_replication`,
+      # so here it would be `geo_cool_widget_replication`
+      def self.replication_enabled_by_default?
+        false
+      end
+
+      override :verification_feature_flag_enabled?
+      def self.verification_feature_flag_enabled?
+        # We are adding verification at the same time as replication, so we
+        # don't need to toggle verification separately from replication. When
+        # the replication feature flag is off, then verification is also off
+        # (see `VerifiableReplicator.verification_enabled?`)
+        true
+      end
+
+    end
+  end
+  ```
+
+- [ ] Generate the feature flag definition file by running the feature flag command and following the command prompts:
+
+  ```shell
+  bin/feature-flag --ee geo_cool_widget_replication --type development --group 'group::geo'
+  ```
+
+- [ ] Add this replicator class to the method `replicator_classes` in
+  `ee/lib/gitlab/geo.rb`:
+
+  ```ruby
+  REPLICATOR_CLASSES = [
+    ::Geo::PackageFileReplicator,
+    ::Geo::CoolWidgetReplicator
+  ]
+  end
+  ```
+
+- [ ] Create `ee/spec/replicators/geo/cool_widget_replicator_spec.rb` and perform the necessary setup to define the `model_record` variable for the shared examples:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe Geo::CoolWidgetReplicator do
+    let(:model_record) { build(:cool_widget) }
+
+    include_examples 'a blob replicator'
+    include_examples 'a verifiable replicator'
+  end
+  ```
+
+- [ ] Create `ee/app/models/geo/cool_widget_registry.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  class Geo::CoolWidgetRegistry < Geo::BaseRegistry
+    include ::Geo::ReplicableRegistry
+    include ::Geo::VerifiableRegistry
+
+    MODEL_CLASS = ::CoolWidget
+    MODEL_FOREIGN_KEY = :cool_widget_id
+
+    belongs_to :cool_widget, class_name: 'CoolWidget'
+  end
+  ```
+
+- [ ] Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`.
+- [ ] Update `def model_class_factory_name` in `ee/spec/services/geo/registry_consistency_service_spec.rb`.
+- [ ] Update `it 'creates missing registries for each registry class'` in `ee/spec/workers/geo/secondary/registry_consistency_worker_spec.rb`.
+- [ ] Add `cool_widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`.
+- [ ] Create `ee/spec/factories/geo/cool_widget_registry.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  FactoryBot.define do
+    factory :geo_cool_widget_registry, class: 'Geo::CoolWidgetRegistry' do
+      cool_widget
+      state { Geo::CoolWidgetRegistry.state_value(:pending) }
+
+      trait :synced do
+        state { Geo::CoolWidgetRegistry.state_value(:synced) }
+        last_synced_at { 5.days.ago }
+      end
+
+      trait :failed do
+        state { Geo::CoolWidgetRegistry.state_value(:failed) }
+        last_synced_at { 1.day.ago }
+        retry_count { 2 }
+        last_sync_failure { 'Random error' }
+      end
+
+      trait :started do
+        state { Geo::CoolWidgetRegistry.state_value(:started) }
+        last_synced_at { 1.day.ago }
+        retry_count { 0 }
+      end
+    end
+  end
+  ```
+
+- [ ] Create `ee/spec/models/geo/cool_widget_registry_spec.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe Geo::CoolWidgetRegistry, :geo, type: :model do
+    let_it_be(:registry) { create(:geo_cool_widget_registry) }
+
+    specify 'factory is valid' do
+      expect(registry).to be_valid
+    end
+
+    include_examples 'a Geo framework registry'
+    include_examples 'a Geo verifiable registry'
+  end
+  ```
+
+#### Step 2. Implement metrics gathering
+
+Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in `GeoNodeStatus` for display in the UI, and sent to Prometheus:
+
+- [ ] Add the following fields to Geo Node Status example responses in `doc/api/geo_nodes.md`:
+  - `cool_widgets_count`
+  - `cool_widgets_checksum_total_count`
+  - `cool_widgets_checksummed_count`
+  - `cool_widgets_checksum_failed_count`
+  - `cool_widgets_synced_count`
+  - `cool_widgets_failed_count`
+  - `cool_widgets_registry_count`
+  - `cool_widgets_verification_total_count`
+  - `cool_widgets_verified_count`
+  - `cool_widgets_verification_failed_count`
+  - `cool_widgets_synced_in_percentage`
+  - `cool_widgets_verified_in_percentage`
+- [ ] Add the same fields to `GET /geo_nodes/status` example response in
+  `ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json`.
+- [ ] Add the following fields to the `Sidekiq metrics` table in `doc/administration/monitoring/prometheus/gitlab_metrics.md`:
+  - `geo_cool_widgets`
+  - `geo_cool_widgets_checksum_total`
+  - `geo_cool_widgets_checksummed`
+  - `geo_cool_widgets_checksum_failed`
+  - `geo_cool_widgets_synced`
+  - `geo_cool_widgets_failed`
+  - `geo_cool_widgets_registry`
+  - `geo_cool_widgets_verification_total`
+  - `geo_cool_widgets_verified`
+  - `geo_cool_widgets_verification_failed`
+- [ ] Add the following to the parameterized table in the `context 'Replicator stats' do` block in `ee/spec/models/geo_node_status_spec.rb`:
+
+  ```ruby
+  Geo::CoolWidgetReplicator | :cool_widget | :geo_cool_widget_registry
+  ```
+
+- [ ] Add the following to `spec/factories/cool_widgets.rb`:
+
+  ```ruby
+  trait(:verification_succeeded) do
+    with_file
+    verification_checksum { 'abc' }
+    verification_state { CoolWidget.verification_state_value(:verification_succeeded) }
+  end
+
+  trait(:verification_failed) do
+    with_file
+    verification_failure { 'Could not calculate the checksum' }
+    verification_state { CoolWidget.verification_state_value(:verification_failed) }
+  end
+  ```
+
+- [ ] Make sure the factory also allows setting a `project` attribute. If the model does not have a direct relation to a project, you can use a `transient` attribute. Check out `spec/factories/merge_request_diffs.rb` for an example.
+
+Cool Widget replication and verification metrics should now be available in the API, the `Admin > Geo > Nodes` view, and Prometheus.
+
+#### Step 3. Implement the GraphQL API
+
+The GraphQL API is used by `Admin > Geo > Replication Details` views, and is directly queryable by administrators.
+
+- [ ] Add a new field to `GeoNodeType` in `ee/app/graphql/types/geo/geo_node_type.rb`:
+
+  ```ruby
+  field :cool_widget_registries, ::Types::Geo::CoolWidgetRegistryType.connection_type,
+        null: true,
+        resolver: ::Resolvers::Geo::CoolWidgetRegistriesResolver,
+        description: 'Find Cool Widget registries on this Geo node',
+        feature_flag: :geo_cool_widget_replication
+  ```
+
+- [ ] Add the new `cool_widget_registries` field name to the `expected_fields` array in `ee/spec/graphql/types/geo/geo_node_type_spec.rb`.
+- [ ] Create `ee/app/graphql/resolvers/geo/cool_widget_registries_resolver.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  module Resolvers
+    module Geo
+      class CoolWidgetRegistriesResolver < BaseResolver
+        type ::Types::Geo::GeoNodeType.connection_type, null: true
+
+        include RegistriesResolver
+      end
+    end
+  end
+  ```
+
+- [ ] Create `ee/spec/graphql/resolvers/geo/cool_widget_registries_resolver_spec.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe Resolvers::Geo::CoolWidgetRegistriesResolver do
+    it_behaves_like 'a Geo registries resolver', :geo_cool_widget_registry
+  end
+  ```
+
+- [ ] Create `ee/app/finders/geo/cool_widget_registry_finder.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  module Geo
+    class CoolWidgetRegistryFinder
+      include FrameworkRegistryFinder
+    end
+  end
+  ```
+
+- [ ] Create `ee/spec/finders/geo/cool_widget_registry_finder_spec.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe Geo::CoolWidgetRegistryFinder do
+    it_behaves_like 'a framework registry finder', :geo_cool_widget_registry
+  end
+  ```
+
+- [ ] Create `ee/app/graphql/types/geo/cool_widget_registry_type.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  module Types
+    module Geo
+      # rubocop:disable Graphql/AuthorizeTypes because it is included
+      class CoolWidgetRegistryType < BaseObject
+        include ::Types::Geo::RegistryType
+
+        graphql_name 'CoolWidgetRegistry'
+        description 'Represents the Geo replication and verification state of a cool_widget'
+
+        field :cool_widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Cool Widget'
+      end
+    end
+  end
+  ```
+
+- [ ] Create `ee/spec/graphql/types/geo/cool_widget_registry_type_spec.rb`:
+
+  ```ruby
+  # frozen_string_literal: true
+
+  require 'spec_helper'
+
+  RSpec.describe GitlabSchema.types['CoolWidgetRegistry'] do
+    it_behaves_like 'a Geo registry type'
+
+    it 'has the expected fields (other than those included in RegistryType)' do
+      expected_fields = %i[cool_widget_id]
+
+      expect(described_class).to have_graphql_fields(*expected_fields).at_least
+    end
+  end
+  ```
+
+- [ ] Add integration tests for providing CoolWidget registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in `ee/spec/requests/api/graphql/geo/registries_spec.rb`:
+
+  ```ruby
+  it_behaves_like 'gets registries for', {
+    field_name: 'coolWidgetRegistries',
+    registry_class_name: 'CoolWidgetRegistry',
+    registry_factory: :geo_cool_widget_registry,
+    registry_foreign_key_field_name: 'coolWidgetId'
+  }
+  ```
+
+- [ ] Update the GraphQL reference documentation:
+
+  ```shell
+  bundle exec rake gitlab:graphql:compile_docs
+  ```
+
+Individual Cool Widget replication and verification data should now be available via the GraphQL API.
+
+### Release Geo support of Cool Widgets
+
+- [ ] In the rollout issue you created when creating the feature flag, modify the Roll Out Steps:
+  - [ ] Cross out any steps related to testing on production GitLab.com, because Geo is not running on production GitLab.com at the moment.
+  - [ ] Add a step to `Test replication and verification of Cool Widgets on a non-GDK-deployment. For example, using GitLab Environment Toolkit`.
+  - [ ] Add a step to `Ping the Geo PM and EM to coordinate testing`. For example, you might add steps to generate Cool Widgets, and then a Geo engineer may take it from there.
+- [ ] In `ee/config/feature_flags/development/geo_cool_widget_replication.yml`, set `default_enabled: true`
+
+- [ ] In `ee/app/replicators/geo/cool_widget_replicator.rb`, delete the `self.replication_enabled_by_default?` method:
+
+  ```ruby
+  module Geo
+    class CoolWidgetReplicator < Gitlab::Geo::Replicator
+      ...
+
+      # REMOVE THIS METHOD
+      def self.replication_enabled_by_default?
+        false
+      end
+      # REMOVE THIS METHOD
+
+      ...
+    end
+  end
+  ```
+
+- [ ] In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `feature_flag` option for the released type:
+
+  ```ruby
+  field :cool_widget_registries, ::Types::Geo::CoolWidgetRegistryType.connection_type,
+        null: true,
+        resolver: ::Resolvers::Geo::CoolWidgetRegistriesResolver,
+        description: 'Find Cool Widget registries on this Geo node',
+        feature_flag: :geo_cool_widget_replication # REMOVE THIS LINE
+  ```
+
+- [ ] Add a row for Cool Widgets to the `Data types` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#data-types)
+- [ ] Add a row for Cool Widgets to the `Limitations on replication/verification` table in [Geo data types support](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/administration/geo/replication/datatypes.md#limitations-on-replicationverification). If the row already exists, then update it to show that Replication and Verification is released in the current version.

doc/development/geo/framework.md

@@ -160,868 +160,16 @@ the Geo team if you are unsure.
 
 ### Blob Replicator Strategy
 
-Models that use
-[CarrierWave's](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base`
-can be easily supported by Geo with the `Geo::BlobReplicatorStrategy` module.
+Models that use [CarrierWave's](https://github.com/carrierwaveuploader/carrierwave) `Uploader::Base` are supported by Geo with the `Geo::BlobReplicatorStrategy` module. For example, see how [Geo replication was implemented for Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/issues/238464).
 
-First, each file should have its own primary ID and model. Geo strongly
-recommends treating *every single file* as a first-class citizen, because in
-our experience this greatly simplifies tracking replication and verification
-state.
+Each file is expected to have its own primary ID and model. Geo strongly recommends treating *every single file* as a first-class citizen, because in our experience this greatly simplifies tracking replication and verification state.
 
-For example, to add support for files referenced by a `Widget` model with a
-`widgets` table, you would perform the following steps:
-
-#### Replication
-
-1. Include `Gitlab::Geo::ReplicableModel` in the `Widget` class, and specify
-   the Replicator class `with_replicator Geo::WidgetReplicator`.
-
-   At this point the `Widget` class should look like this:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class Widget < ApplicationRecord
-     include ::Gitlab::Geo::ReplicableModel
-
-     with_replicator Geo::WidgetReplicator
-
-     mount_uploader :file, WidgetUploader
-
-     # @param primary_key_in [Range, Widget] arg to pass to primary_key_in scope
-     # @return [ActiveRecord::Relation<Widget>] everything that should be synced to this node, restricted by primary key
-     def self.replicables_for_current_secondary(primary_key_in)
-       # Should be implemented. The idea of the method is to restrict
-       # the set of synced items depending on synchronization settings
-     end
-     ...
-   end
-   ```
-
-   If there is a common constraint for records to be available for replication,
-   make sure to also overwrite the `available_replicables` scope.
-
-1. Create `ee/app/replicators/geo/widget_replicator.rb`. Implement the
-   `#carrierwave_uploader` method which should return a `CarrierWave::Uploader`,
-   and implement the class method `.model` to return the `Widget` class:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   module Geo
-     class WidgetReplicator < Gitlab::Geo::Replicator
-       include ::Geo::BlobReplicatorStrategy
-
-       def self.model
-         ::Widget
-       end
-
-       def carrierwave_uploader
-         model_record.file
-       end
-
-       # The feature flag follows the format `geo_#{replicable_name}_replication`,
-       # so here it would be `geo_widget_replication`
-       def self.replication_enabled_by_default?
-         false
-       end
-     end
-   end
-   ```
-
-1. Add this replicator class to the method `replicator_classes` in
-   `ee/lib/gitlab/geo.rb`:
-
-   ```ruby
-   REPLICATOR_CLASSES = [
-      ::Geo::PackageFileReplicator,
-      ::Geo::WidgetReplicator
-   ]
-   end
-   ```
-
-1. Create `ee/spec/replicators/geo/widget_replicator_spec.rb` and perform
-   the necessary setup to define the `model_record` variable for the shared
-   examples:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   require 'spec_helper'
-
-   RSpec.describe Geo::WidgetReplicator do
-     let(:model_record) { build(:widget) }
-
-     it_behaves_like 'a blob replicator'
-   end
-   ```
-
-1. Create the `widget_registry` table, with columns ordered according to [our guidelines](../ordering_table_columns.md) so Geo secondaries can track the sync and
-   verification state of each Widget's file. This migration belongs in `ee/db/geo/migrate`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class CreateWidgetRegistry < ActiveRecord::Migration[6.0]
-     include Gitlab::Database::MigrationHelpers
-
-     disable_ddl_transaction!
-
-     def up
-       unless table_exists?(:widget_registry)
-         ActiveRecord::Base.transaction do
-           create_table :widget_registry, id: :bigserial, force: :cascade do |t|
-             t.bigint :widget_id, null: false
-             t.datetime_with_timezone :created_at, null: false
-             t.datetime_with_timezone :last_synced_at
-             t.datetime_with_timezone :retry_at
-             t.datetime_with_timezone :verified_at
-             t.datetime_with_timezone :verification_started_at
-             t.datetime_with_timezone :verification_retry_at
-             t.integer :state, default: 0, null: false, limit: 2
-             t.integer :verification_state, default: 0, null: false, limit: 2
-             t.integer :retry_count, default: 0, limit: 2
-             t.integer :verification_retry_count, default: 0, limit: 2
-             t.boolean :checksum_mismatch, default: false, null: false
-             t.binary :verification_checksum
-             t.binary :verification_checksum_mismatched
-             t.string :verification_failure, limit: 255
-             t.string :last_sync_failure, limit: 255
-
-             t.index :widget_id, name: :index_widget_registry_on_widget_id, unique: true
-             t.index :retry_at
-             t.index :state
-             # To optimize performance of WidgetRegistry.verification_failed_batch
-             t.index :verification_retry_at, name:  :widget_registry_failed_verification, order: "NULLS FIRST",  where: "((state = 2) AND (verification_state = 3))"
-             # To optimize performance of WidgetRegistry.needs_verification_count
-             t.index :verification_state, name:  :widget_registry_needs_verification, where: "((state = 2)  AND (verification_state = ANY (ARRAY[0, 3])))"
-             # To optimize performance of WidgetRegistry.verification_pending_batch
-             t.index :verified_at, name: :widget_registry_pending_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 0))"
-           end
-         end
-       end
-     end
-
-     def down
-       drop_table :widget_registry
-     end
-   end
-   ```
-
-1. Create `ee/app/models/geo/widget_registry.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class Geo::WidgetRegistry < Geo::BaseRegistry
-     include ::Geo::ReplicableRegistry
-     include ::Geo::VerifiableRegistry
-
-     MODEL_CLASS = ::Widget
-     MODEL_FOREIGN_KEY = :widget_id
-
-     belongs_to :widget, class_name: 'Widget'
-   end
-   ```
-
-1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`.
-1. Add `widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`.
-1. Create `ee/spec/factories/geo/widget_registry.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   FactoryBot.define do
-     factory :geo_widget_registry, class: 'Geo::WidgetRegistry' do
-       widget
-       state { Geo::WidgetRegistry.state_value(:pending) }
-
-       trait :synced do
-         state { Geo::WidgetRegistry.state_value(:synced) }
-         last_synced_at { 5.days.ago }
-       end
-
-       trait :failed do
-         state { Geo::WidgetRegistry.state_value(:failed) }
-         last_synced_at { 1.day.ago }
-         retry_count { 2 }
-         last_sync_failure { 'Random error' }
-       end
-
-       trait :started do
-         state { Geo::WidgetRegistry.state_value(:started) }
-         last_synced_at { 1.day.ago }
-         retry_count { 0 }
-       end
-     end
-   end
-   ```
-
-1. Create `ee/spec/models/geo/widget_registry_spec.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   require 'spec_helper'
-
-   RSpec.describe Geo::WidgetRegistry, :geo, type: :model do
-     let_it_be(:registry) { create(:geo_widget_registry) }
-
-     specify 'factory is valid' do
-       expect(registry).to be_valid
-     end
-
-     include_examples 'a Geo framework registry'
-     include_examples 'a Geo verifiable registry'
-
-     describe '.find_registry_differences' do
-       ... # To be implemented
-     end
-   end
-   ```
-
-Widgets should now be replicated by Geo.
-
-#### Verification
-
-There are two ways to add verification related fields so that the Geo primary
-can track verification state.
-
-##### Option 1: Add verification state fields to the existing `widgets` table itself
-
-1. Add a migration to add columns ordered according to [our guidelines](../ordering_table_columns.md)
-   for verification state to the widgets table:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class AddVerificationStateToWidgets < ActiveRecord::Migration[6.0]
-     def change
-       change_table(:widgets) do |t|
-         t.integer :verification_state, default: 0, limit: 2, null: false
-         t.column :verification_started_at, :datetime_with_timezone
-         t.integer :verification_retry_count, limit: 2
-         t.column :verification_retry_at, :datetime_with_timezone
-         t.column :verified_at, :datetime_with_timezone
-         t.binary :verification_checksum, using: 'verification_checksum::bytea'
-
-         # rubocop:disable Migration/AddLimitToTextColumns
-         t.text :verification_failure
-         # rubocop:enable Migration/AddLimitToTextColumns
-       end
-     end
-   end
-   ```
-
-1. Adding a `text` column also [requires](../database/strings_and_the_text_data_type.md#add-a-text-column-to-an-existing-table)
-   setting a limit:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class AddVerificationFailureLimitToWidgets < ActiveRecord::Migration[6.0]
-     include Gitlab::Database::MigrationHelpers
-
-     disable_ddl_transaction!
-
-     CONSTRAINT_NAME = 'widget_verification_failure_text_limit'
-
-     def up
-       add_text_limit :widget, :verification_failure, 255, constraint_name: CONSTRAINT_NAME
-     end
-
-     def down
-       remove_check_constraint(:widget, CONSTRAINT_NAME)
-     end
-   end
-   ```
-
-1. Add indexes on verification fields to ensure verification can be performed efficiently:
-
-   Some or all of these indexes can be omitted if the table is guaranteed to be small. Ask a database expert if you are unsure.
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class AddVerificationIndexesToWidgets < ActiveRecord::Migration[6.0]
-     include Gitlab::Database::MigrationHelpers
-
-     PENDING_VERIFICATION_INDEX_NAME = "index_widgets_pending_verification"
-     FAILED_VERIFICATION_INDEX_NAME = "index_widgets_failed_verification"
-     NEEDS_VERIFICATION_INDEX_NAME = "index_widgets_needs_verification"
-
-     disable_ddl_transaction!
-
-     def up
-       add_concurrent_index :widgets, :verification_state, name: VERIFICATION_STATE_INDEX_NAME
-       add_concurrent_index :widgets, :verified_at, where: "(verification_state = 0)", order: { verified_at: 'ASC NULLS FIRST' }, name: PENDING_VERIFICATION_INDEX_NAME
-       add_concurrent_index :widgets, :verification_retry_at, where: "(verification_state = 3)", order: { verification_retry_at: 'ASC NULLS FIRST' }, name: FAILED_VERIFICATION_INDEX_NAME
-       add_concurrent_index :widgets, :verification_state, where: "(verification_state = 0 OR verification_state = 3)", name: NEEDS_VERIFICATION_INDEX_NAME
-     end
-
-     def down
-       remove_concurrent_index_by_name :widgets, VERIFICATION_STATE_INDEX_NAME
-       remove_concurrent_index_by_name :widgets, PENDING_VERIFICATION_INDEX_NAME
-       remove_concurrent_index_by_name :widgets, FAILED_VERIFICATION_INDEX_NAME
-       remove_concurrent_index_by_name :widgets, NEEDS_VERIFICATION_INDEX_NAME
-     end
-   end
-   ```
-
-1. Add the `Gitlab::Geo::VerificationState` concern to the `widget` model if it is not already included in `Gitlab::Geo::ReplicableModel`:
-
-   ```ruby
-   class Widget < ApplicationRecord
-     ...
-     include ::Gitlab::Geo::VerificationState
-     ...
-   end
-   ```
-
-##### Option 2: Create a separate `widget_states` table with verification state fields
-
-1. Create a `widget_states` table and add an index on `verification_state` to ensure verification can be performed efficiently. Order the columns according to [the guidelines](../ordering_table_columns.md):
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class CreateWidgetStates < ActiveRecord::Migration[6.0]
-     include Gitlab::Database::MigrationHelpers
-
-     disable_ddl_transaction!
-
-     def up
-       unless table_exists?(:widget_states)
-         with_lock_retries do
-           create_table :widget_states, id: false do |t|
-             t.references :widget, primary_key: true, null: false, foreign_key: { on_delete: :cascade }
-             t.integer :verification_state, default: 0, limit: 2, null: false
-             t.column :verification_started_at, :datetime_with_timezone
-             t.datetime_with_timezone :verification_retry_at
-             t.datetime_with_timezone :verified_at
-             t.integer :verification_retry_count, limit: 2
-             t.binary :verification_checksum, using: 'verification_checksum::bytea'
-             t.text :verification_failure
-
-             t.index :verification_state, name: "index_widget_states_on_verification_state"
-           end
-         end
-       end
-
-       add_text_limit :widget_states, :verification_failure, 255
-     end
-
-     def down
-       drop_table :widget_states
-     end
-   end
-   ```
-
-1. Add the following lines to the `widget_state.rb` model:
-
-   ```ruby
-   class WidgetState < ApplicationRecord
-     ...
-     self.primary_key = :widget_id
-
-     include ::Gitlab::Geo::VerificationState
-
-     belongs_to :widget, inverse_of: :widget_state
-     ...
-   end
-   ```
-
-1. Add the following lines to the `widget` model:
-
-   ```ruby
-   class Widget < ApplicationRecord
-     ...
-     has_one :widget_state, inverse_of: :widget
-
-     delegate :verification_retry_at, :verification_retry_at=,
-              :verified_at, :verified_at=,
-              :verification_checksum, :verification_checksum=,
-              :verification_failure, :verification_failure=,
-              :verification_retry_count, :verification_retry_count=,
-              to: :widget_state
-     ...
-   end
-   ```
-
-To do: Add verification on secondaries. This should be done as part of
-[Geo: Self Service Framework - First Implementation for Package File verification](https://gitlab.com/groups/gitlab-org/-/epics/1817)
-
-Widgets should now be verified by Geo.
-
-#### Metrics
-
-Metrics are gathered by `Geo::MetricsUpdateWorker`, persisted in
-`GeoNodeStatus` for display in the UI, and sent to Prometheus:
-
-1. Add fields `widgets_count`, `widgets_checksummed_count`,
-   `widgets_checksum_failed_count`, `widgets_synced_count`,
-   `widgets_failed_count`, and `widgets_registry_count` to
-   `GET /geo_nodes/status` example response in
-   `doc/api/geo_nodes.md`.
-1. Add the same fields to `GET /geo_nodes/status` example response in
-   `ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json`.
-1. Add fields `geo_widgets`, `geo_widgets_checksummed`,
-   `geo_widgets_checksum_failed`, `geo_widgets_synced`,
-   `geo_widgets_failed`, and `geo_widgets_registry` to
-   `Sidekiq metrics` table in
-   `doc/administration/monitoring/prometheus/gitlab_metrics.md`.
-1. Add the following to the parameterized table in
-   `ee/spec/models/geo_node_status_spec.rb`:
-
-   ```ruby
-   Geo::WidgetReplicator | :widget | :geo_widget_registry
-   ```
-
-1. Add the following to `spec/factories/widgets.rb`:
-
-   ```ruby
-   trait(:verification_succeeded) do
-     with_file
-     verification_checksum { 'abc' }
-     verification_state { Widget.verification_state_value(:verification_succeeded) }
-   end
-
-   trait(:verification_failed) do
-     with_file
-     verification_failure { 'Could not calculate the checksum' }
-     verification_state { Widget.verification_state_value(:verification_failed) }
-   end
-   ```
-
-1. Make sure the factory also allows setting a `project` attribute. If the model
-   does not have a direct relation to a project, you can use a `transient`
-   attribute. Check out `spec/factories/merge_request_diffs.rb` for an example.
-
-Widget replication and verification metrics should now be available in the API,
-the Admin Area UI, and Prometheus.
-
-#### GraphQL API
-
-1. Add a new field to `GeoNodeType` in
-   `ee/app/graphql/types/geo/geo_node_type.rb`:
-
-   ```ruby
-   field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type,
-         null: true,
-         resolver: ::Resolvers::Geo::WidgetRegistriesResolver,
-         description: 'Find widget registries on this Geo node',
-         feature_flag: :geo_widget_replication
-   ```
-
-1. Add the new `widget_registries` field name to the `expected_fields` array in
-   `ee/spec/graphql/types/geo/geo_node_type_spec.rb`.
-1. Create `ee/app/graphql/resolvers/geo/widget_registries_resolver.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   module Resolvers
-     module Geo
-       class WidgetRegistriesResolver < BaseResolver
-         include RegistriesResolver
-       end
-     end
-   end
-   ```
-
-1. Create `ee/spec/graphql/resolvers/geo/widget_registries_resolver_spec.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   require 'spec_helper'
-
-   RSpec.describe Resolvers::Geo::WidgetRegistriesResolver do
-     it_behaves_like 'a Geo registries resolver', :geo_widget_registry
-   end
-   ```
-
-1. Create `ee/app/finders/geo/widget_registry_finder.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   module Geo
-     class WidgetRegistryFinder
-       include FrameworkRegistryFinder
-     end
-   end
-   ```
-
-1. Create `ee/spec/finders/geo/widget_registry_finder_spec.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   require 'spec_helper'
-
-   RSpec.describe Geo::WidgetRegistryFinder do
-     it_behaves_like 'a framework registry finder', :geo_widget_registry
-   end
-   ```
-
-1. Create `ee/app/graphql/types/geo/widget_registry_type.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   module Types
-     module Geo
-       # rubocop:disable Graphql/AuthorizeTypes because it is included
-       class WidgetRegistryType < BaseObject
-         include ::Types::Geo::RegistryType
-
-         graphql_name 'WidgetRegistry'
-         description 'Represents the Geo sync and verification state of a widget'
-
-         field :widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Widget'
-       end
-     end
-   end
-   ```
-
-1. Create `ee/spec/graphql/types/geo/widget_registry_type_spec.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   require 'spec_helper'
-
-   RSpec.describe GitlabSchema.types['WidgetRegistry'] do
-     it_behaves_like 'a Geo registry type'
-
-     it 'has the expected fields (other than those included in RegistryType)' do
-       expected_fields = %i[widget_id]
-
-       expect(described_class).to have_graphql_fields(*expected_fields).at_least
-     end
-   end
-   ```
-
-1. Add integration tests for providing Widget registry data to the frontend via
-   the GraphQL API, by duplicating and modifying the following shared examples
-   in `ee/spec/requests/api/graphql/geo/registries_spec.rb`:
-
-   ```ruby
-   it_behaves_like 'gets registries for', {
-     field_name: 'widgetRegistries',
-     registry_class_name: 'WidgetRegistry',
-     registry_factory: :geo_widget_registry,
-     registry_foreign_key_field_name: 'widgetId'
-   }
-   ```
-
-1. Update the GraphQL reference documentation:
-
-   ```shell
-   bundle exec rake gitlab:graphql:compile_docs
-   ```
-
-Individual widget synchronization and verification data should now be available
-via the GraphQL API.
-
-Make sure to replicate the "update" events. Geo Framework does not currently support
-replicating "update" events because all entities added to the framework, by this time,
-are immutable. If this is the case
-for the entity you're going to add, follow <https://gitlab.com/gitlab-org/gitlab/-/issues/118743>
-and <https://gitlab.com/gitlab-org/gitlab/-/issues/118745> as examples to add the new event type.
-Also, remove this notice when you've added it.
-
-#### Admin UI
-
-To do: This should be done as part of
-[Geo: Implement frontend for Self-Service Framework replicables](https://gitlab.com/groups/gitlab-org/-/epics/2525)
-
-Widget sync and verification data (aggregate and individual) should now be
-available in the Admin UI.
-
-#### Releasing the feature
-
-1. In `ee/config/feature_flags/development/geo_widget_replication.yml`, set `default_enabled: true`
-
-1. In `ee/app/replicators/geo/widget_replicator.rb`, delete the `self.replication_enabled_by_default?` method:
-
-   ```ruby
-   module Geo
-     class WidgetReplicator < Gitlab::Geo::Replicator
-       ...
-
-       # REMOVE THIS METHOD
-       def self.replication_enabled_by_default?
-         false
-       end
-       # REMOVE THIS METHOD
-
-       ...
-     end
-   end
-   ```
-
-1. In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `feature_flag` option for the released type:
-
-   ```ruby
-   field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type,
-         null: true,
-         resolver: ::Resolvers::Geo::WidgetRegistriesResolver,
-         description: 'Find widget registries on this Geo node',
-         feature_flag: :geo_widget_replication # REMOVE THIS LINE
-   ```
+To implement Geo replication of a new blob-type Model, [open an issue with the provided issue template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Geo%3A%20Replicate%20a%20new%20blob%20type).
 
 ### Repository Replicator Strategy
 
-Models that refer to any repository on the disk
-can be easily supported by Geo with the `Geo::RepositoryReplicatorStrategy` module.
-
-For example, to add support for files referenced by a `Gizmos` model with a
-`gizmos` table, you would perform the following steps.
-
-#### Replication
-
-1. Include `Gitlab::Geo::ReplicableModel` in the `Gizmo` class, and specify
-   the Replicator class `with_replicator Geo::GizmoReplicator`.
-
-   At this point the `Gizmo` class should look like this:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class Gizmo < ApplicationRecord
-     include ::Gitlab::Geo::ReplicableModel
-
-     with_replicator Geo::GizmoReplicator
-
-     # @param primary_key_in [Range, Gizmo] arg to pass to primary_key_in scope
-     # @return [ActiveRecord::Relation<Gizmo>] everything that should be synced to this node, restricted by primary key
-     def self.replicables_for_current_secondary(primary_key_in)
-       # Should be implemented. The idea of the method is to restrict
-       # the set of synced items depending on synchronization settings
-     end
-
-     # Geo checks this method in FrameworkRepositorySyncService to avoid
-     # snapshotting repositories using object pools
-     def pool_repository
-       nil
-     end
-     ...
-   end
-   ```
-
-   Pay some attention to method `pool_repository`. Not every repository type uses
-   repository pooling. As Geo prefers to use repository snapshotting, it can lead to data loss.
-   Make sure to overwrite `pool_repository` so it returns nil for repositories that do not
-   have pools.
-
-   If there is a common constraint for records to be available for replication,
-   make sure to also overwrite the `available_replicables` scope.
-
-1. Create `ee/app/replicators/geo/gizmo_replicator.rb`. Implement the
-   `#repository` method which should return a `<Repository>` instance,
-   and implement the class method `.model` to return the `Gizmo` class:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   module Geo
-     class GizmoReplicator < Gitlab::Geo::Replicator
-       include ::Geo::RepositoryReplicatorStrategy
-
-       def self.model
-         ::Gizmo
-       end
-
-       def repository
-         model_record.repository
-       end
-
-       def self.git_access_class
-         ::Gitlab::GitAccessGizmo
-       end
-
-       # The feature flag follows the format `geo_#{replicable_name}_replication`,
-       # so here it would be `geo_gizmo_replication`
-       def self.replication_enabled_by_default?
-         false
-       end
-     end
-   end
-   ```
-
-1. Generate the feature flag definition file by running the feature flag command
-   and running through the steps:
-
-   ```shell
-   bin/feature-flag --ee geo_gizmo_replication --type development --group 'group::geo'
-   ```
-
-1. Make sure Geo push events are created. Usually it needs some
-   change in the `app/workers/post_receive.rb` file. Example:
-
-   ```ruby
-   def replicate_gizmo_changes(gizmo)
-     if ::Gitlab::Geo.primary?
-       gizmo.replicator.handle_after_update if gizmo
-     end
-   end
-   ```
-
-   See `app/workers/post_receive.rb` for more examples.
-
-1. Make sure the repository removal is also handled. You may need to add something
-   like the following in the destroy service of the repository:
-
-   ```ruby
-   gizmo.replicator.handle_after_destroy if gizmo.repository
-   ```
-
-1. Add this replicator class to the method `replicator_classes` in
-   `ee/lib/gitlab/geo.rb`:
-
-   ```ruby
-   REPLICATOR_CLASSES = [
-      ...
-      ::Geo::PackageFileReplicator,
-      ::Geo::GizmoReplicator
-   ]
-   end
-   ```
-
-1. Create `ee/spec/replicators/geo/gizmo_replicator_spec.rb` and perform
-   the necessary setup to define the `model_record` variable for the shared
-   examples:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   require 'spec_helper'
-
-   RSpec.describe Geo::GizmoReplicator do
-     let(:model_record) { build(:gizmo) }
-
-     include_examples 'a repository replicator'
-   end
-   ```
-
-1. Create the `gizmo_registry` table, with columns ordered according to [our guidelines](../ordering_table_columns.md) so Geo secondaries can track the sync and
-   verification state of each Gizmo. This migration belongs in `ee/db/geo/migrate`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class CreateGizmoRegistry < ActiveRecord::Migration[6.0]
-     include Gitlab::Database::MigrationHelpers
-
-     disable_ddl_transaction!
-
-     def up
-       create_table :gizmo_registry, id: :bigserial, force: :cascade do |t|
-         t.datetime_with_timezone :retry_at
-         t.datetime_with_timezone :last_synced_at
-         t.datetime_with_timezone :created_at, null: false
-         t.bigint :gizmo_id, null: false
-         t.integer :state, default: 0, null: false, limit: 2
-         t.integer :retry_count, default: 0, limit: 2
-         t.string :last_sync_failure, limit: 255
-         t.boolean :force_to_redownload
-         t.boolean :missing_on_primary
-
-         t.index :gizmo_id, name: :index_gizmo_registry_on_gizmo_id, unique: true
-         t.index :retry_at
-         t.index :state
-        end
-      end
-
-      def down
-        drop_table :gizmo_registry
-      end
-   end
-   ```
-
-1. Create `ee/app/models/geo/gizmo_registry.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   class Geo::GizmoRegistry < Geo::BaseRegistry
-     include Geo::ReplicableRegistry
-
-     MODEL_CLASS = ::Gizmo
-     MODEL_FOREIGN_KEY = :gizmo_id
-
-     belongs_to :gizmo, class_name: 'Gizmo'
-   end
-   ```
-
-1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`.
-1. Add `gizmo_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`.
-1. Create `ee/spec/factories/geo/gizmo_registry.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   FactoryBot.define do
-     factory :geo_gizmo_registry, class: 'Geo::GizmoRegistry' do
-       gizmo
-       state { Geo::GizmoRegistry.state_value(:pending) }
-
-       trait :synced do
-         state { Geo::GizmoRegistry.state_value(:synced) }
-         last_synced_at { 5.days.ago }
-       end
-
-       trait :failed do
-         state { Geo::GizmoRegistry.state_value(:failed) }
-         last_synced_at { 1.day.ago }
-         retry_count { 2 }
-         last_sync_failure { 'Random error' }
-       end
-
-       trait :started do
-         state { Geo::GizmoRegistry.state_value(:started) }
-         last_synced_at { 1.day.ago }
-         retry_count { 0 }
-       end
-     end
-   end
-   ```
-
-1. Create `ee/spec/models/geo/gizmo_registry_spec.rb`:
-
-   ```ruby
-   # frozen_string_literal: true
-
-   require 'spec_helper'
-
-   RSpec.describe Geo::GizmoRegistry, :geo, type: :model do
-     let_it_be(:registry) { create(:geo_gizmo_registry) }
-
-     specify 'factory is valid' do
-       expect(registry).to be_valid
-     end
-
-     include_examples 'a Geo framework registry'
-   end
-   ```
-
-1. Make sure the newly added repository type can be accessed by a secondary.
-   You may need to make some changes to one of the Git access classes.
-
-   Gizmos should now be replicated by Geo.
-
-#### Metrics
-
-You need to make the same changes as for Blob Replicator Strategy.
-You need to make the same changes for the [metrics as in the Blob Replicator Strategy](#metrics).
-
-#### GraphQL API
-
-You need to make the same changes for the GraphQL API [as in the Blob Replicator Strategy](#graphql-api).
+Models that refer to any Git repository on disk are supported by Geo with the `Geo::RepositoryReplicatorStrategy` module. For example, see how [Geo replication was implemented for Group-level Wikis](https://gitlab.com/gitlab-org/gitlab/-/issues/208147). Note that this issue does not implement verification, since verification of Git repositories was not yet added to the Geo self-service framework. An example implementing verification can be found in the merge request to [Add Snippet repository verification](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56596).
 
-#### Releasing the feature
+Each Git repository is expected to have its own primary ID and model.
 
-You need to make the same changes for [releasing the feature as in the Blob Replicator Strategy](#releasing-the-feature).
+To implement Geo replication of a new Git repository-type Model, [open an issue with the provided issue template](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Geo%3A%20Replicate%20a%20new%20Git%20repository%20type).