BugLink: https://bugs.launchpad.net/bugs/1703742

The current configuration is set to always use transparent hugepages
by default. There exists plenty of anecdotal evidence that this is
less than perfect a choice and in some scenarios it leads to some
performance issues.

My own investigations with stress-ng stream and malloc tests show that
the current default impacts performance. I ran various test scenarios
on different MADVISE configurations, each result below is based on
the average of 5 runs on an i7-3770 CPU @ 3.4GHz with 8GB memory,
8MB L3 cache, 256K L2 cache, 32K/32K L1 cache.

All the above results are from an average of 5 rounds of tests.

malloc allocation stressor:

     malloc     always    madvise
    size (MB)   ops/sec   ops/sec
         32     1254.43   2422.49
         64     2100.36   4300.28
        128     3768.57   7215.38
        256     7940.73  14893.85
        512    17618.62  26861.29
       1024    32777.17  48029.37

Clearly madvise is more performent.

stream bandwidth/compute stressor:

    stream      always    madvise
                         NOHUGEPAGE
    size (MB)   MB/sec     MB/sec
          1   17713.54   18439.69
          2   12460.34   13015.46
          4   12195.81   12694.51
          8   12085.11   12674.26
         16   12054.09   12649.00
         32   12082.42   12409.65
         64   12262.88   12084.85
        128   12235.25   11788.49
        256   11808.69   11283.69
        512   11970.01   12434.82

For small allocations, always is less performant. Large
allocations can enable the more performant transparent
huge pages with madvise(2) if we disable always as default.

Other stress-ng memory allocation/writing/freeing and madvise
operations showed little significant differences.

I have also experimented with boot testing Ubuntu with kernels
configured with different MADVISE configs and found there is
little noticeable difference in performance, so I believe that
there is little scope for any kitten killer performance regressions
with this change.

This change will by default not use transparent huge pages unless
madvise(2) is used to instruct the kernel to do so on a memory
mapping.  According to the madvise(2) manual, this only takes
effect on private anonymous mappings with MADV_HUGEPAGE.
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Acked-by: Kamal Mostafa <kamal@canonical.com>
Acked-by: Stefan Bader <stefan.bader@canonical.com>
Signed-off-by: Khalid Elmously <khalid.elmously@canonical.com>