1. 24 Aug, 2021 1 commit
    • Yorick Peterse's avatar
      Steal pending merge request diff commit user jobs · b98a2d0c
      Yorick Peterse authored
      The background migration MigrateMergeRequestDiffCommitUsers processes
      batches of merge_request_diff_commits rows in batches. These migrations
      have been running for several weeks now.
      
      Unfortunately, on GitLab.com we are running into problems with the
      remaining 750 or so jobs. Some jobs take a few minutes to complete,
      while others can take over an hour to complete. The scheduling interval
      was originally just a few minutes. This lead to many jobs being thrown
      away, as they could not be picked up fast enough when there is a long
      running job. Rescheduling jobs with a greater interval hasn't proven
      successful.
      
      In this commit we add a new background migration to clean up the pending
      work. This background migration picks up a single job (in reverse
      order), processes it, then schedules the next job. This approach
      prevents jobs from conflicting with each other if they take too long.
      
      In addition, we change the existing migration to not process jobs that
      have already been processed. This way existing jobs that have yet to be
      picked up won't lead to duplicate work.
      
      Both the old and new migration _can_ process work in parallel, as they
      are different background migration classes. This shouldn't pose any
      problems. For the last few weeks we've been manually running jobs in a
      Rails console in parallel to the Sidekiq jobs, without any problems
      surfacing.
      
      See https://gitlab.com/gitlab-org/gitlab/-/issues/334394 for more
      information.
      
      Changelog: added
      b98a2d0c
  2. 23 Aug, 2021 39 commits