Merge branch 'osw-add-docs-for-idempotent-workers' into 'master'

Add docs for idempotent workers

Closes gitlab-com/gl-infra/scalability#179

See merge request gitlab-org/gitlab!25551

doc/development/sidekiq_style_guide.md

@@ -64,6 +64,63 @@ the extra jobs will take resources away from jobs from workers that were already
 there, if the resources available to the Sidekiq process handling the namespace
 are not adjusted appropriately.
 
+## Idempotent Jobs
+
+It's known that a job can fail for multiple reasons, for example, network outages or bugs.
+In order to address this, Sidekiq has a built-in retry mechanism that is
+used by default by most workers within GitLab.
+
+It's expected that a job can run again after a failure without major side-effects for the
+application or users, which is why Sidekiq encourages
+jobs to be [idempotent and transactional](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional).
+
+As a general rule, a worker can be considered idempotent if:
+
+- It can safely run multiple times with the same arguments.
+- Application side-effects are expected to happen only once
+  (or side-effects of a second run are not impactful).
+
+A good example of that would be a cache expiration worker.
+
+### Ensuring a worker is idempotent
+
+Make sure the worker tests pass using the following shared example:
+
+```ruby
+include_examples 'an idempotent worker' do
+  it 'marks the MR as merged' do
+    # Using subject inside this block will process the job multiple times
+    subject
+
+    expect(merge_request.state).to eq('merged')
+  end
+end
+```
+
+Use the `perform_multiple` method directly instead of `job.perform` (this
+helper method is automatically included for workers).
+
+### Declaring a worker as idempotent
+
+```ruby
+class IdempotentWorker
+  include ApplicationWorker
+
+  # Declares a worker is idempotent and can
+  # safely run multiple times.
+  idempotent!
+
+  # ...
+end
+```
+
+It's encouraged to only have the `idempotent!` call in the top-most worker class, even if
+the `perform` method is defined in another class or module.
+
+NOTE: **Note:**
+Note that a cop will fail if the worker class is not marked as idempotent.
+Consider skipping the cop if you're not confident your job can safely run multiple times.
+
 ## Latency Sensitive Jobs
 
 If a large number of background jobs get scheduled at once, queueing of jobs may