• Dylan Griffith's avatar
    Improve performance of Elasticsearch notes permissions migration · 9abb6fb5
    Dylan Griffith authored
    We are currently running this migration in production to reindex all the
    notes in GitLab.com. There happen to be over 30M of them so it will take
    quite a while to run. We have found in
    https://gitlab.com/gitlab-org/gitlab/-/issues/324745 that the migration
    is not performing as well as we'd hoped. Given that we want to keep each
    job taking less than 1 minute to run we want to reduce the batch size to
    make that easier.
    
    We have found that 9000 per 3 minutes is too much and we are regularly
    overlapping with the previous batch and thus we end up often processing
    the same batch twice.
    
    In addition it will be more efficient to push the work into the "initial
    indexing queue" instead of our main "incremental indexing queue" as the
    initial indexing queue has far less traffic (it only covers newly
    created projects while incremental is any record update in any project).
    9abb6fb5
make-notes-migration-more-efficient.yml 123 Bytes