Merge branch 'mk-compute-total-mem-use' into 'master'

Add mem_total_bytes metric to Memory instrumentation

See merge request gitlab-org/gitlab!61794

doc/development/performance.md

@@ -365,14 +365,16 @@ This patch is available by default for
 [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit/-/merge_requests/149)
 and can additionally be enabled for [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/advanced.md#apply-custom-patches-for-ruby).
 
-This patch provides a set of 3 metrics that makes it easier to understand efficiency of memory usage for a given codepath:
+This patch provides the following metrics that make it easier to understand efficiency of memory use for a given codepath:
 
+- `mem_total_bytes`: the number of bytes consumed both due to new objects being allocated into existing object slots
+                     plus additional memory allocated for large objects (that is, `mem_bytes + slot_size * mem_objects`).
+- `mem_bytes`: the number of bytes allocated by `malloc` for objects that did not fit into an existing object slot.
 - `mem_objects`: the number of objects allocated.
-- `mem_bytes`: the number of bytes allocated by malloc.
-- `mem_mallocs`: the number of malloc allocations.
+- `mem_mallocs`: the number of `malloc` calls.
 
 The number of objects and bytes allocated impact how often GC cycles happen.
-Fewer objects allocations result in a significantly more responsive application.
+Fewer object allocations result in a significantly more responsive application.
 
 It is advised that web server requests do not allocate more than `100k mem_objects`
 and `100M mem_bytes`. You can view the current usage on [GitLab.com](https://log.gprd.gitlab.net/goto/3a9678bb595e3f89a0c7b5c61bcc47b9).

lib/gitlab/memory/instrumentation.rb

@@ -45,9 +45,12 @@ module Gitlab
       end
 
       # This method returns a hash with the following keys:
-      # - mem_objects: a number of allocated heap slots (as reflected by GC)
-      # - mem_mallocs: a number of malloc calls
-      # - mem_bytes: a number of bytes allocated with a mallocs tied to heap slots
+      # - mem_objects:     number of allocated heap slots (as reflected by GC)
+      # - mem_mallocs:     number of malloc calls
+      # - mem_bytes:       number of bytes allocated by malloc for objects that did not fit
+      #                    into a heap slot
+      # - mem_total_bytes: number of bytes allocated for both objects consuming an object slot
+      #                    and objects that required a malloc (mem_malloc_bytes)
       def self.measure_thread_memory_allocations(previous)
         return unless available?
         return unless previous
@@ -56,9 +59,13 @@ module Gitlab
         return unless current
 
         # calculate difference in a memory allocations
-        previous.to_h do |key, value|
+        result = previous.to_h do |key, value|
           [KEY_MAPPING.fetch(key), current[key].to_i - value]
         end
+
+        result[:mem_total_bytes] = result[:mem_bytes] + result[:mem_objects] * GC::INTERNAL_CONSTANTS[:RVALUE_SIZE]
+
+        result
       end
 
       def self.with_memory_allocations

spec/lib/gitlab/memory/instrumentation_spec.rb

@@ -69,10 +69,12 @@ RSpec.describe Gitlab::Memory::Instrumentation do
       end
 
       it 'a hash is returned' do
-        is_expected.to include(
+        result = subject
+        expect(result).to include(
           mem_objects: be > 1000,
           mem_mallocs: be > 1000,
-          mem_bytes: be > 100_000 # 100 items * 100 bytes each
+          mem_bytes: be > 100_000, # 100 items * 100 bytes each
+          mem_total_bytes: eq(result[:mem_bytes] + 40 * result[:mem_objects])
         )
       end
     end