1. 20 Mar, 2020 1 commit
    • Sean McGivern's avatar
      Disable JobWaiter when there are many jobs · 29132e38
      Sean McGivern authored
      In WaitableWorker, when there are fewer than four jobs to process we
      simply process them without going through Sidekiq. When there are more
      jobs, we run them in Sidekiq but add a second argument for a JobWaiter
      key, and wait for `timeout` seconds to see if those jobs finished.
      
      When there are very many jobs in the batch, we will never finish the
      batch before the timeout. This has these consequences:
      
      1. The HTTP request takes longer as we're waiting for the timeout.
      2. The jobs can't be deduplicated as the second argument is a unique
         JobWaiter key.
      
      This change adds a feature flag, skip_job_waiter_for_large_batches, that
      doesn't use the JobWaiter key when the batch size is more than ten times
      the timeout, as we can process approximately ten jobs per second. By
      default the flag is off.
      29132e38
  2. 19 Mar, 2020 28 commits
  3. 18 Mar, 2020 11 commits