Update self service framework docs

to include options for adding verification state
fields and improve documentation.

doc/development/geo/framework.md

@@ -219,10 +219,30 @@ For example, to add support for files referenced by a `Widget` model with a
        def carrierwave_uploader
          model_record.file
        end
+
+       # Change this to `true` to release replication of this model. Then remove
+       # this override in the next release.
+       # The feature flag follows the format `geo_#{replicable_name}_replication`,
+       # so here it would be `geo_widget_replication`
+       def self.replication_enabled_by_default?
+         false
+       end
      end
    end
    ```
 
+1. Add this replicator class to the method `replicator_classes` in
+`ee/lib/gitlab/geo.rb`:
+
+   ```ruby
+   def self.replicator_classes
+     classes = [::Geo::PackageFileReplicator,
+                ::Geo::WidgetReplicator]
+
+     classes.select(&:enabled?)
+   end
+   ```
+
 1. Create `ee/spec/replicators/geo/widget_replicator_spec.rb` and perform
    the setup necessary to define the `model_record` variable for the shared
    examples.
@@ -239,13 +259,15 @@ For example, to add support for files referenced by a `Widget` model with a
    end
    ```
 
-1. Create the `widget_registry` table so Geo secondaries can track the sync and
+1. Create the `widget_registry` table, with columns ordered according to [our guidelines](../ordering_table_columns.md) so Geo secondaries can track the sync and
    verification state of each Widget's file. This migration belongs in `ee/db/geo/migrate`:
 
    ```ruby
    # frozen_string_literal: true
 
    class CreateWidgetRegistry < ActiveRecord::Migration[6.0]
+     include Gitlab::Database::MigrationHelpers
+
      DOWNTIME = false
 
      disable_ddl_transaction!
@@ -257,12 +279,12 @@ For example, to add support for files referenced by a `Widget` model with a
              t.integer :widget_id, null: false
              t.integer :state, default: 0, null: false, limit: 2
              t.integer :retry_count, default: 0, limit: 2
-             t.text :last_sync_failure
              t.datetime_with_timezone :retry_at
              t.datetime_with_timezone :last_synced_at
              t.datetime_with_timezone :created_at, null: false
+             t.text :last_sync_failure
 
-             t.index :widget_id
+             t.index :widget_id, name: :index_widget_registry_on_widget_id
              t.index :retry_at
              t.index :state
            end
@@ -358,8 +380,13 @@ Widgets should now be replicated by Geo!
 
 #### Verification
 
-1. Add verification state fields to the `widgets` table so the Geo primary can
-   track verification state:
+There are two ways to add verification related fields so that the Geo primary
+can track verification state:
+
+##### Option 1: Add verification state fields to the existing `widgets` table itself
+
+1. Add a migration to add columns ordered according to [our guidelines](../ordering_table_columns.md)
+for verification state to the widgets table:
 
    ```ruby
    # frozen_string_literal: true
@@ -382,7 +409,7 @@ Widgets should now be replicated by Geo!
    end
    ```
 
-   Adding a `text` column also [requires](../database/strings_and_the_text_data_type.md#add-a-text-column-to-an-existing-table)
+1. Adding a `text` column also [requires](../database/strings_and_the_text_data_type.md#add-a-text-column-to-an-existing-table)
    setting a limit:
 
    ```ruby
@@ -400,7 +427,7 @@ Widgets should now be replicated by Geo!
    ```
 
 1. Add a partial index on `verification_failure` and `verification_checksum` to ensure
-   re-verification can be performed efficiently:
+   re-verification can be performed efficiently. Add a migration in `ee/db/geo/migrate/`:
 
    ```ruby
    # frozen_string_literal: true
@@ -424,6 +451,65 @@ Widgets should now be replicated by Geo!
    end
    ```
 
+##### Option 2: Create a separate `widget_states` table with verification state fields
+
+1. Add a migration in `ee/db/geo/migrate/` to create a `widget_states` table and add a
+   partial index on `verification_failure` and `verification_checksum` to ensure
+   re-verification can be performed efficiently. Order the columns according to [our guidelines](../ordering_table_columns.md):
+
+   ```ruby
+   # frozen_string_literal: true
+
+   class CreateWidgetStates < ActiveRecord::Migration[6.0]
+     include Gitlab::Database::MigrationHelpers
+
+     DOWNTIME = false
+
+     disable_ddl_transaction!
+
+     def up
+       unless table_exists?(:widget_states)
+         with_lock_retries do
+           create_table :widget_states, id: false do |t|
+             t.references :widget, primary_key: true, null: false, foreign_key: { on_delete: :cascade }
+             t.datetime_with_timezone :verification_retry_at
+             t.datetime_with_timezone :verified_at
+             t.integer :verification_retry_count, limit: 2
+             t.binary :verification_checksum, using: 'verification_checksum::bytea'
+             t.text :verification_failure
+
+             t.index :verification_failure, where: "(verification_failure IS NOT NULL)", name: "widgets_verification_failure_partial"
+             t.index :verification_checksum, where: "(verification_checksum IS NOT NULL)", name: "widgets_verification_checksum_partial"
+           end
+         end
+       end
+
+       add_text_limit :widget_states, :verification_failure, 255
+     end
+
+     def down
+       drop_table :widget_states
+     end
+   end
+   ```
+
+1. Add the following lines to the `widget` model:
+
+   ```ruby
+   class Widget < ApplicationRecord
+     ...
+     has_one :widget_state, inverse_of: :widget
+
+     delegate :verification_retry_at, :verification_retry_at=,
+              :verified_at, :verified_at=,
+              :verification_checksum, :verification_checksum=,
+              :verification_failure, :verification_failure=,
+              :verification_retry_count, :verification_retry_count=,
+              to: :widget_state
+     ...
+   end
+   ```
+
 To do: Add verification on secondaries. This should be done as part of
 [Geo: Self Service Framework - First Implementation for Package File verification](https://gitlab.com/groups/gitlab-org/-/epics/1817)