Add memory optimisation docs

This extends memory profiling
to be memory optimisation documentation.

This includes a few examples of the different
types of allocations.

doc/development/performance.md

@@ -347,13 +347,112 @@ example, you can find which tests take longest to run or which execute the most
 queries. This can be handy for optimizing our tests or identifying performance
 issues in our code.
 
-## Memory profiling
+## Memory optimization
 
-We can use two approaches, often in combination, to track down memory issues:
+We can use a set of different techniques, often in combination, to track down memory issues:
 
 - Leaving the code intact and wrapping a profiler around it.
+- Use memory allocation counters for requests and services.
 - Monitor memory usage of the process while disabling/enabling different parts of the code we suspect could be problematic.
 
+### Memory allocations
+
+Ruby shipped with GitLab includes a special patch to allow [tracing memory allocations](https://gitlab.com/gitlab-org/gitlab/-/issues/296530).
+This patch is available by default for
+[Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4948),
+[CNG](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/591),
+[GitLab CI](https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/355),
+[GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/merge_requests/149)
+and can additionally be enabled for [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/advanced.md#apply-custom-patches-for-ruby).
+
+This patch provides a set of 3 metrics that makes it easier to understand efficiency of memory usage for a given codepath:
+
+- `mem_objects`: the number of objects allocated.
+- `mem_bytes`: the number of bytes allocated by malloc.
+- `mem_mallocs`: the number of malloc allocations.
+
+The number of objects and bytes allocated impact how often GC cycles happen.
+Fewer objects allocations result in a significantly more responsive application.
+
+It is advised that web server requests do not allocate more than `100k mem_objects`
+and `100M mem_bytes`. You can view the current usage on [GitLab.com](https://log.gprd.gitlab.net/goto/3a9678bb595e3f89a0c7b5c61bcc47b9).
+
+#### Checking memory pressure of own code
+
+There are two ways of measuring your own code:
+
+1. Review `api_json.log`, `development_json.log`, `sidekiq.log` that includes memory allocation counters.
+1. Use `Gitlab::Memory::Instrumentation.with_memory_allocations` for a given codeblock and log it.
+1. Use [Measuring module](service_measurement.md)
+
+```json
+{"time":"2021-02-15T11:20:40.821Z","severity":"INFO","duration_s":0.27412,"db_duration_s":0.05755,"view_duration_s":0.21657,"status":201,"method":"POST","path":"/api/v4/projects/user/1","mem_objects":86705,"mem_bytes":4277179,"mem_mallocs":22693,"correlation_id":"...}
+```
+
+#### Different types of allocations
+
+The `mem_*` values represent different aspects of how objects and memory are allocated in Ruby:
+
+- The following example will create around of `1000` of `mem_objects` since strings
+   can be frozen, and while the underlying string object remains the same, we still need to allocate 1000 references to this string:
+
+  ```ruby
+  Gitlab::Memory::Instrumentation.with_memory_allocations do
+    1_000.times { '0123456789' }
+  end
+
+  => {:mem_objects=>1001, :mem_bytes=>0, :mem_mallocs=>0}
+  ```
+
+- The following example will create around of `1000` of `mem_objects`, as strings are created dynamically.
+   Each of them will not allocate additional memory, as they fit into Ruby slot of 40 bytes:
+
+  ```ruby
+  Gitlab::Memory::Instrumentation.with_memory_allocations do
+    s = '0'
+    1_000.times { s * 23 }
+  end
+
+  => {:mem_objects=>1002, :mem_bytes=>0, :mem_mallocs=>0}
+  ```
+
+- The following example will create around of `1000` of `mem_objects`, as strings are created dynamically.
+   Each of them will allocate additional memory as strings are larger than Ruby slot of 40 bytes:
+
+  ```ruby
+  Gitlab::Memory::Instrumentation.with_memory_allocations do
+    s = '0'
+    1_000.times { s * 24 }
+  end
+
+  => {:mem_objects=>1002, :mem_bytes=>32000, :mem_mallocs=>1000}
+  ```
+
+- The following example will allocate over 40kB of data, and perform only a single memory allocation.
+   The existing object will be reallocated/resized on subsequent iterations:
+
+  ```ruby
+  Gitlab::Memory::Instrumentation.with_memory_allocations do
+    str = ''
+    append = '0123456789012345678901234567890123456789' # 40 bytes
+    1_000.times { str.concat(append) }
+  end
+  => {:mem_objects=>3, :mem_bytes=>49152, :mem_mallocs=>1}
+  ```
+
+- The following example will create over 1k of objects, perform over 1k of allocations, each time mutating the object.
+   This does result in copying a lot of data and perform a lot of memory allocations
+  (as represented by `mem_bytes` counter) indicating very inefficient method of appending string:
+
+  ```ruby
+  Gitlab::Memory::Instrumentation.with_memory_allocations do
+    str = ''
+    append = '0123456789012345678901234567890123456789' # 40 bytes
+    1_000.times { str += append }
+  end
+  => {:mem_objects=>1003, :mem_bytes=>21968752, :mem_mallocs=>1000}
+  ```
+
 ### Using Memory Profiler
 
 We can use `memory_profiler` for profiling.