1. 24 Aug, 2020 1 commit
    • Stan Hu's avatar
      Fix missing pipeline e-mails when job logs moved to object storage · 3a2ba7b1
      Stan Hu authored
      As discussed in https://gitlab.com/gitlab-org/gitlab/-/issues/195430,
      it's possible that a race condition occurs when a build finishes and a
      failed pipeline email goes out. This is what was happening before:
      
      1. Build finishes and causes a pipeline failure. The pipeline failure
      transition causes the pipeline to run `PipelineNotificationWorker`,
      which schedules an `ActiveJob` to e-mail a user with a failed pipeline
      ID.
      
      2. `BuildFinishedWorker` schedules `ArchiveTraceWorker` to move the job
      log, stored locally on a filesystem, to object storage.
      
      3. The `ActiveJob` runs and loads the pipeline and failed build
      logs. Some builds have been moved to object storage, but the `ActiveJob`
      has a stale record and attempts to load the file from the filesystem.
      
      4. The file doesn't exist, an `Errno::ENOENT` exception is raised.
      
      To fix this, we now attempt a refresh the job from the database if we
      encounter this exception.
      3a2ba7b1
  2. 22 Aug, 2020 5 commits
  3. 21 Aug, 2020 34 commits