• Stan Hu's avatar
    Fix ActiveRecord::IrreversibleOrderError during restore from backup · fc715de9
    Stan Hu authored
    Due to the frequency with which settings are accessed, it is likely that
    during a backup restore a running GitLab process will insert a new
    `application_settings` row before the constraints have been added to the
    table. This would add an extra row with ID 1 and prevent the primary key
    constraint from being added, which made ActiveRecord throw a
    IrreversibleOrderError anytime the settings were accessed.
    
    The following sequence was seen in PostgreSQL logs:
    
    1. Restore: CREATE TABLE public.application_settings ...
    2. Restore: CREATE SEQUENCE public.application_settings_id_seq
    3. Puma/Sidekiq: INSERT INTO "application_settings" ...
    4. Restore: COPY public.application_settings (id, ...
    5. Restore: ADD CONSTRAINT application_settings_pkey PRIMARY KEY (id);
    
    In step 3, since GitLab inserted a new row with ID 1, but shortly after
    that the restore process would also copy another row with ID 1. As a
    result, the ADD CONSTRAINT in step 5 would fail with the message:
    
    ERROR:   could not create unique index "application_settings_pkey"
    Key (id)=(1) is duplicated.
    
    The right way to fix this would be to restore the database in a single
    transaction via the `--single-transaction` flag in `psql`, but this
    usually fails due to permission errors when extensions and schemas are
    dropped and recreated. `psql` generally works best with superuser
    access, but restore usually runs with limited access permissions.
    
    GitLab normally inserts an `application_settings` row if no row is
    found, but to prevent duplicate rows from being inserted we ensure that
    a primary key has been set beforehand. This should help prevent primary
    key collisions and duplicate rows from being added.
    
    We still need to tell administrators to stop all processes that connect
    to the database (puma, sidekiq, and gitlab-exporter), but this check
    should ensure that a primary key constraint is added during a restore.
    
    Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/36405
    fc715de9
sh-fix-backup-restore-race.yml 120 Bytes