1. 08 Nov, 2019 22 commits
    • Michael Anthony Knyszek's avatar
      runtime: add page cache and tests · 81640ea3
      Michael Anthony Knyszek authored
      This change adds a page cache structure which owns a chunk of free pages
      at a given base address. It also adds code to allocate to this cache
      from the page allocator. Finally, it adds tests for both.
      
      Notably this change does not yet integrate the code into the runtime,
      just into runtime tests.
      
      Updates #35112.
      
      Change-Id: Ibe121498d5c3be40390fab58a3816295601670df
      Reviewed-on: https://go-review.googlesource.com/c/go/+/196643
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      81640ea3
    • Udalov Max's avatar
      encoding/binary: make Read return an error when data is not a pointer · c444ec30
      Udalov Max authored
      Make binary.Read return an error when passed `data` argument is not
      a pointer to a fixed-size value or a slice of fixed-size values.
      
      Fixes #32927
      
      Change-Id: I04f48be55fe9b0cc66c983d152407d0e42cbcd95
      Reviewed-on: https://go-review.googlesource.com/c/go/+/184957Reviewed-by: default avatarBrad Fitzpatrick <bradfitz@golang.org>
      c444ec30
    • Bryan C. Mills's avatar
      cmd/go/internal/lockedfile: add a unit-test for Transform · a84ac189
      Bryan C. Mills authored
      Updates #35425
      
      Change-Id: I9ca2251246ee2fa9bb7a335d5eff94d3c9f1f004
      Reviewed-on: https://go-review.googlesource.com/c/go/+/206143Reviewed-by: default avatarBrad Fitzpatrick <bradfitz@golang.org>
      a84ac189
    • Bryan C. Mills's avatar
      cmd/go/internal/modload: use lockedfile.Read for the initial read of the go.mod file · c5fac1ed
      Bryan C. Mills authored
      Updates #34634
      Fixes #35425
      
      Change-Id: I878a8d229b33dcde9e7d4dfd82ddf9815d38a465
      Reviewed-on: https://go-review.googlesource.com/c/go/+/206142
      Run-TryBot: Bryan C. Mills <bcmills@google.com>
      Reviewed-by: default avatarBrad Fitzpatrick <bradfitz@golang.org>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      c5fac1ed
    • Michael Anthony Knyszek's avatar
      runtime: add per-p mspan cache · 4517c02f
      Michael Anthony Knyszek authored
      This change adds a per-p mspan object cache similar to the sudog cache.
      Unfortunately this cache can't quite operate like the sudog cache, since
      it is used in contexts where write barriers are disallowed (i.e.
      allocation codepaths), so rather than managing an array and a slice,
      it's just an array and a length. A little bit more unsafe, but avoids
      any write barriers.
      
      The purpose of this change is to reduce the number of operations which
      require the heap lock in allocation, paving the way for a lockless fast
      path.
      
      Updates #35112.
      
      Change-Id: I32cfdcd8528fb7be985640e4f3a13cb98ffb7865
      Reviewed-on: https://go-review.googlesource.com/c/go/+/196642
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      4517c02f
    • Michael Anthony Knyszek's avatar
      runtime: rearrange mheap_.alloc* into allocSpan · a762221b
      Michael Anthony Knyszek authored
      This change combines the functionality of allocSpanLocked, allocManual,
      and alloc_m into a new method called allocSpan. While these methods'
      abstraction boundaries are OK when the heap lock is held throughout,
      they start to break down when we want finer-grained locking in the page
      allocator.
      
      allocSpan does just that, and only locks the heap when it absolutely has
      to. Piggy-backing off of work in previous CLs to make more of span
      initialization lockless, this change makes span initialization entirely
      lockless as part of the reorganization.
      
      Ultimately this change will enable us to add a lockless fast path to
      allocSpan.
      
      Updates #35112.
      
      Change-Id: I99875939d75fb4e958a67ac99e4a7cda44f06864
      Reviewed-on: https://go-review.googlesource.com/c/go/+/196641
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      a762221b
    • Michael Anthony Knyszek's avatar
      runtime: fix (*gcSweepBuf).block guarantees · a5a6f610
      Michael Anthony Knyszek authored
      Currently gcSweepBuf guarantees that push operations may be performed
      concurrently with each other and that block operations may be performed
      concurrently with push operations as well.
      
      Unfortunately, this isn't quite true. The existing code allows push
      operations to happen concurrently with each other, but block operations
      may return blocks with nil entries. The way this can happen is if two
      concurrent pushers grab a slot to push to, and the first one (the one
      with the earlier slot in the buffer) doesn't quite write a span value
      when the block is called. The existing code in block only checks if the
      very last value in the block is nil, when really an arbitrary number of
      the last few values in the block may or may not be nil.
      
      Today, this case can't actually happen because when push operations
      happen concurrently during a GC (which is the only time block is
      called), they only ever happen during an allocation with the heap lock
      held, effectively serializing them. A block operation may happen
      concurrently with one of these pushes, but its callers will never see a
      nil mspan. Outside of a GC, this isn't a problem because although push
      operations from allocations can run concurrently with push operations
      from sweeping, block operations will never run.
      
      In essence, the real concurrency guarantees provided by gcSweepBuf are
      that block operations may happen concurrently with push operations, but
      that push operations may not be concurrent with each other if there are
      any block operations.
      
      To fix this, and to prepare for push operations happening without the
      heap lock held in a future CL, we update the documentation for block to
      correctly state that there may be nil entries in the returned slice.
      While we're here, make the mspan writes into the buffer atomic to avoid
      a block user racing on a nil check, and document that the user should
      load mspan values from the returned slice atomically. Finally, we make
      all callers of block adhere to the new rules.
      
      We choose to allow nil values rather than filter them out because the
      only caller of block is markrootSpans, and if it catches a nil entry,
      then there wasn't anything to mark in there anyway since the span is
      just being created.
      
      Updates #35112.
      
      Change-Id: I6450aab15f51690d7a000ba5b3d529cf2ca5da1e
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203318
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      a5a6f610
    • Michael Anthony Knyszek's avatar
      runtime: make more page sweeper operations atomic · dac936a4
      Michael Anthony Knyszek authored
      This change makes it so that allocation and free related page sweeper
      metadata operations (e.g. pageInUse and pagesInUse) are atomic rather
      than protected by the heap lock. This will help in reducing the length
      of the critical path with the heap lock held in future changes.
      
      Updates #35112.
      
      Change-Id: Ie82bff024204dd17c4c671af63350a7a41add354
      Reviewed-on: https://go-review.googlesource.com/c/go/+/196640
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      dac936a4
    • Cherry Zhang's avatar
      cmd/internal/obj/arm64: make function epilogue async-signal safe · 47232f0d
      Cherry Zhang authored
      When the frame size is large, we generate
      
      MOVD.P	0xf0(SP), LR
      ADD	$(framesize-0xf0), SP
      
      This is problematic: after the first instruction, we have a
      partial frame of size (framesize-0xf0). If we try to unwind the
      stack at this point, we'll try to read the LR from the stack at
      0(SP) (the new SP) as the frame size is not 0. But this slot does
      not contain a valid LR.
      
      Fix this by not changing SP in two instructions. Instead,
      generate
      
      MOVD	(SP), LR
      ADD	$framesize, SP
      
      This affects not only async preemption but also profiling. So we
      change the generated instructions, instead of marking unsafe
      point.
      
      Change-Id: I4e78c62d50ffc4acff70ccfbfec16a5ccae17f24
      Reviewed-on: https://go-review.googlesource.com/c/go/+/206057
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarIan Lance Taylor <iant@golang.org>
      47232f0d
    • Cherry Zhang's avatar
      runtime: add async preemption support on PPC64 · 374c2847
      Cherry Zhang authored
      This CL adds support of call injection and async preemption on
      PPC64.
      
      For the injected call to return to the preempted PC, we have to
      clobber either LR or CTR. For reasons mentioned in previous CLs,
      we choose CTR. Previous CLs have marked code sequences that use
      CTR async-nonpreemtible.
      
      Change-Id: Ia642b5f06a890dd52476f45023b2a830c522eee0
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203824
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      374c2847
    • Michael Anthony Knyszek's avatar
      runtime: remove unnecessary large parameter to mheap_.alloc · 7f574e47
      Michael Anthony Knyszek authored
      mheap_.alloc currently accepts both a spanClass and a "large" parameter
      indicating whether the allocation is large. These are redundant, since
      spanClass.sizeclass() == 0 is an equivalent way to determine this and is
      already used in mheap_.alloc. There are no places in the runtime where
      the size class could be non-zero and large == true.
      
      Updates #35112.
      
      Change-Id: Ie66facf8f0faca6f4cd3d20a8ac4bc259e11823d
      Reviewed-on: https://go-review.googlesource.com/c/go/+/196639
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      7f574e47
    • Michael Anthony Knyszek's avatar
      runtime: define maximum supported physical page and huge page sizes · ffb5646f
      Michael Anthony Knyszek authored
      This change defines a maximum supported physical and huge page size in
      the runtime based on the new page allocator's implementation, and uses
      them where appropriate.
      
      Furthemore, if the system exceeds the maximum supported huge page
      size, we simply ignore it silently.
      
      It also fixes a huge-page-related test which is only triggered by a
      condition which is definitely wrong.
      
      Finally, it adds a few TODOs related to code clean-up and supporting
      larger huge page sizes.
      
      Updates #35112.
      Fixes #35431.
      
      Change-Id: Ie4348afb6bf047cce2c1433576d1514720d8230f
      Reviewed-on: https://go-review.googlesource.com/c/go/+/205937
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      Reviewed-by: default avatarCherry Zhang <cherryyz@google.com>
      ffb5646f
    • Bryan C. Mills's avatar
      cmd/go: delete flaky TestQEMUUserMode · a782472d
      Bryan C. Mills authored
      If QEMU user-mode is actually a supported configuration, then per
      http://golang.org/wiki/PortingPolicy it needs to have a builder
      running tests for all packages, not just a simple “hello world”
      program.
      
      Updates #1508
      Updates #13024
      Fixes #35457
      
      Change-Id: Ib6122b06ad1d265550a0e92131506266495893cc
      Reviewed-on: https://go-review.googlesource.com/c/go/+/206137
      Run-TryBot: Bryan C. Mills <bcmills@google.com>
      Reviewed-by: default avatarBrad Fitzpatrick <bradfitz@golang.org>
      a782472d
    • Michael Anthony Knyszek's avatar
      runtime: ensure heap memstats are updated atomically · ae4534e6
      Michael Anthony Knyszek authored
      For the most part, heap memstats are already updated atomically when
      passed down to OS-level memory functions (e.g. sysMap). Elsewhere,
      however, they're updated with the heap lock.
      
      In order to facilitate holding the heap lock for less time during
      allocation paths, this change more consistently makes the update of
      these statistics atomic by calling mSysStat{Inc,Dec} appropriately
      instead of simply adding or subtracting. It also ensures these values
      are loaded atomically.
      
      Furthermore, an undocumented but safe update condition for these
      memstats is during STW, at which point using atomics is unnecessary.
      This change also documents this condition in mstats.go.
      
      Updates #35112.
      
      Change-Id: I87d0b6c27b98c88099acd2563ea23f8da1239b66
      Reviewed-on: https://go-review.googlesource.com/c/go/+/196638
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      ae4534e6
    • Michael Anthony Knyszek's avatar
      runtime: remove useless heap_objects accounting · 814c5058
      Michael Anthony Knyszek authored
      This change removes useless additional heap_objects accounting for large
      objects. heap_objects is computed from scratch at ReadMemStats time
      (which stops the world) by using nlargealloc and nlargefree, so mutating
      heap_objects turns out to be pointless.
      
      As a result, the "large" parameter on "mheap_.freeSpan" is no longer
      necessary and so this change cleans that up too.
      
      Change-Id: I7d6b486d9b57c018e3db46221d81b55fe4c1b021
      Reviewed-on: https://go-review.googlesource.com/c/go/+/196637
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      814c5058
    • Michael Anthony Knyszek's avatar
      runtime: make allocNeedsZero lock-free · 4208dbef
      Michael Anthony Knyszek authored
      In preparation for a lockless fast path in the page allocator, this
      change makes it so that checking if an allocation needs to be zeroed may
      be done atomically.
      
      Unfortunately, this means there is a CAS-loop to ensure monotonicity of
      the zeroedBase value in heapArena. This CAS-loop exits if an allocator
      acquiring memory further on in the arena wins or if it succeeds. The
      CAS-loop should have a relatively small amount of contention because of
      this monotonicity, though it would be ideal if we could just have
      CAS-ers with the greatest value always win. The CAS-loop is unnecessary
      in the steady-state, but should bring some start-up performance gains as
      it's likely cheaper than the additional zeroing required, especially for
      large allocations.
      
      For very large allocations that span arenas, the CAS-loop should be
      completely uncontended for most of the arenas it touches, it may only
      encounter contention on the first and last arena.
      
      Updates #35112.
      
      Change-Id: If3d19198b33f1b1387b71e1ce5902d39a5c0f98e
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203859
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      4208dbef
    • Bryan C. Mills's avatar
      runtime: skip TestPingPongHog on builders · 52aebe8d
      Bryan C. Mills authored
      This test is failing consistently in the longtest builders,
      potentially masking regressions in other packages.
      
      Updates #35271
      
      Change-Id: Idc03171c0109b5c8d4913e0af2078c1115666897
      Reviewed-on: https://go-review.googlesource.com/c/go/+/206098Reviewed-by: default avatarCarlos Amedee <carlos@golang.org>
      52aebe8d
    • Hana (Hyang-Ah) Kim's avatar
      runtime/pprof: delete unused locForPC · 45b4ed75
      Hana (Hyang-Ah) Kim authored
      Change-Id: Ie4754fefba6057b1cf558d0096fe0e83355f8eff
      Reviewed-on: https://go-review.googlesource.com/c/go/+/205098
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      45b4ed75
    • Hana (Hyang-Ah) Kim's avatar
      runtime/pprof: correct inlined function location encoding for non-CPU profiles · e25de44e
      Hana (Hyang-Ah) Kim authored
      Change-Id: Id270a3477bf1a581755c4311eb12f990aa2260b5
      Reviewed-on: https://go-review.googlesource.com/c/go/+/205097
      Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      e25de44e
    • Ian Lance Taylor's avatar
      Revert "math/cmplx: handle special cases" · f5e89c22
      Ian Lance Taylor authored
      This reverts CL 169501.
      
      Reason for revert: The new tests fail at least on s390x and MIPS. This is likely a minor bug in the compiler or runtime. But this point in the release cycle is not the time to debug these details, which are unlikely to be new. Let's try again for 1.15.
      
      Updates #29320
      Fixes #35443
      
      Change-Id: I2218b2083f8974b57d528e3742524393fc72b355
      Reviewed-on: https://go-review.googlesource.com/c/go/+/206037
      Run-TryBot: Ian Lance Taylor <iant@golang.org>
      Reviewed-by: default avatarBryan C. Mills <bcmills@google.com>
      Reviewed-by: default avatarBrad Fitzpatrick <bradfitz@golang.org>
      f5e89c22
    • Hana (Hyang-Ah) Kim's avatar
      runtime/pprof: correctly encode inlined functions in CPU profile · e038c7e4
      Hana (Hyang-Ah) Kim authored
      The pprof profile proto message expects inlined functions of a PC
      to be encoded in one Location entry using multiple Line entries.
      https://github.com/google/pprof/blob/5e96527/proto/profile.proto#L177-L184
      
      runtime/pprof has encoded the symbolization information by creating
      a Location for each PC found in the stack trace and including info
      from all the frames expanded from the PC using runtime.CallersFrames.
      This assumes inlined functions are represented as a single PC in the
      stack trace. (https://go-review.googlesource.com/41256)
      
      In the recent years, behavior around inlining and the traceback
      changed significantly (e.g. https://golang.org/cl/152537,
      https://golang.org/issue/29582, and many changes). Now the PCs
      in the stack trace represent user frames even including inline
      marks. As a result, the profile proto started to allocate a Location
      entry for each user frame, lose the inline information (so pprof
      presented incorrect results when inlined functions are involved),
      and confuse the pprof tool with those PCs made up for inline marks.
      
      This CL attempts to detect inlined call frames from the stack traces
      of CPU profiles, and organize the Location information as intended.
      Currently, runtime does not provide a reliable and convenient way to
      detect inlined call frames and expand user frames from a given externally
      recognizable PCs. So we use heuristics to recover the groups
        - inlined call frames have nil Func field
        - inlined call frames will have the same Entry point
        - but must be careful with recursive functions that have the
          same Entry point by definition, and non-Go functions that
          may lack most of the fields of Frame.
      
      The followup CL will address the issue with other profile types.
      
      Change-Id: I0c9667ab016a3e898d648f31c3f82d84c15398db
      Reviewed-on: https://go-review.googlesource.com/c/go/+/204636Reviewed-by: default avatarKeith Randall <khr@golang.org>
      e038c7e4
    • Michael Anthony Knyszek's avatar
      runtime: remove old page allocator · 33dfd352
      Michael Anthony Knyszek authored
      This change removes the old page allocator from the runtime.
      
      Updates #35112.
      
      Change-Id: Ib20e1c030f869b6318cd6f4288a9befdbae1b771
      Reviewed-on: https://go-review.googlesource.com/c/go/+/195700
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      33dfd352
  2. 07 Nov, 2019 18 commits
    • Michael Anthony Knyszek's avatar
      runtime: switch to new page allocator · e6135c27
      Michael Anthony Knyszek authored
      This change flips the oldPageAllocator constant enabling the new page
      allocator in the Go runtime.
      
      Updates #35112.
      
      Change-Id: I7fc8332af9fd0e43ce28dd5ebc1c1ce519ce6d0c
      Reviewed-on: https://go-review.googlesource.com/c/go/+/201765
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      e6135c27
    • Cherry Zhang's avatar
      runtime: disable async preemption on darwin/arm(64) for now · 52d5e76b
      Cherry Zhang authored
      Enabling async preemption on darwin/arm and darwin/arm64 causes
      the builder to fail, e.g.
      https://build.golang.org/log/03f727b8f91b0c75bf54ff508d7d2f00b5cad4bf
      
      Due to the limited resource, I haven't been able to get access on
      those devices to debug. Disable async preemption for now.
      
      Updates #35439.
      
      Change-Id: I5a31ad6962c2bae8e6e9b8303c494610a8a4e50a
      Reviewed-on: https://go-review.googlesource.com/c/go/+/205842Reviewed-by: default avatarBrad Fitzpatrick <bradfitz@golang.org>
      52d5e76b
    • Keith Randall's avatar
      doc: document new math.Fma function · 953cc749
      Keith Randall authored
      This accidentally got committed - please review the whole paragraph
      as if it was new.
      
      Change-Id: I98e1db4670634c6e792d26201ce0cd329a6928b6
      Reviewed-on: https://go-review.googlesource.com/c/go/+/202579Reviewed-by: default avatarIan Lance Taylor <iant@golang.org>
      953cc749
    • Matthew Dempsky's avatar
      cmd/compile: restore more missing -m=2 escape analysis details · 4cde749f
      Matthew Dempsky authored
      This CL also restores analysis details for (1) expressions that are
      directly heap allocated because of being too large for the stack or
      non-constant in size, and (2) for assignments that we short circuit
      because we flow their address to another escaping object.
      
      No change to normal compilation behavior. Only adds additional Printfs
      guarded by -m=2.
      
      Updates #31489.
      
      Change-Id: I43682195d389398d75ced2054e29d9907bb966e7
      Reviewed-on: https://go-review.googlesource.com/c/go/+/205917
      Run-TryBot: Matthew Dempsky <mdempsky@google.com>
      Reviewed-by: default avatarDavid Chase <drchase@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      4cde749f
    • Cherry Zhang's avatar
      runtime: add async preemption support on MIPS and MIPS64 · a930fede
      Cherry Zhang authored
      This CL adds support of call injection and async preemption on
      MIPS and MIPS64.
      
      Like ARM64, we need to clobber one register (REGTMP) for
      returning from the injected call. Previous CLs have marked code
      sequences that use REGTMP async-nonpreemtible.
      
      It seems on MIPS/MIPS64, a CALL instruction is not "atomic" (!).
      If a signal is delivered right at the CALL instruction, we may
      see an updated LR with a not-yet-updated PC. In some cases this
      may lead to failed stack unwinding. Don't preempt in this case.
      
      Change-Id: I99437b2d05869ded5c0c8cb55265dbfc933aedab
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203720Reviewed-by: default avatarKeith Randall <khr@golang.org>
      a930fede
    • Michael Anthony Knyszek's avatar
      runtime: compute whether a span needs zeroing in the new page allocator · a120cc8b
      Michael Anthony Knyszek authored
      This change adds the allocNeedZero method to mheap which uses the new
      heapArena field zeroedBase to determine whether a new allocation needs
      zeroing. The purpose of this work is to avoid zeroing memory that is
      fresh from the OS in the context of the new allocator, where we no
      longer have the concept of a free span to track this information.
      
      The new field in heapArena, zeroedBase, is small, which runs counter to
      the advice in the doc comment for heapArena. Since heapArenas are
      already not a multiple of the system page size, this advice seems stale,
      and we're OK with using an extra physical page for a heapArena. So, this
      change also deletes the comment with that advice.
      
      Updates #35112.
      
      Change-Id: I688cd9fd3c57a98a6d43c45cf699543ce16697e2
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203858
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      a120cc8b
    • Cherry Zhang's avatar
      runtime: add async preemption support on S390X · 933bf75e
      Cherry Zhang authored
      This CL adds support of call injection and async preemption on
      S390X.
      
      Like ARM64, we need to clobber one register (REGTMP) for
      returning from the injected call. Previous CLs have marked code
      sequences that use REGTMP async-nonpreemtible.
      
      Change-Id: I78adbc5fd70ca245da390f6266623385b45c9dfc
      Reviewed-on: https://go-review.googlesource.com/c/go/+/204106Reviewed-by: default avatarKeith Randall <khr@golang.org>
      933bf75e
    • Cherry Zhang's avatar
      cmd/internal/obj/s390x: mark unsafe points · 4751db93
      Cherry Zhang authored
      For async preemption, we will be using REGTMP as a temporary
      register in injected call on S390X, which will clobber it. So any
      code that uses REGTMP is not safe for async preemption.
      
      In the assembler backend, we expand a Prog to multiple machine
      instructions and use REGTMP as a temporary register if necessary.
      These need to be marked unsafe. Unlike ARM64 and MIPS,
      instructions on S390X are variable length so we don't use the
      length as a condition. Instead, we set a bit on the Prog whenever
      REGTMP is used.
      
      Change-Id: Ie5d14068a950f4c7cea51dff2c4a8bdc19ec9348
      Reviewed-on: https://go-review.googlesource.com/c/go/+/204105
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      4751db93
    • Michael Anthony Knyszek's avatar
      runtime: integrate new page allocator into runtime · 689f6f77
      Michael Anthony Knyszek authored
      This change integrates all the bits and pieces of the new page allocator
      into the runtime, behind a global constant.
      
      Updates #35112.
      
      Change-Id: I6696bde7bab098a498ab37ed2a2caad2a05d30ec
      Reviewed-on: https://go-review.googlesource.com/c/go/+/201764
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      689f6f77
    • Michael Anthony Knyszek's avatar
      runtime: make the scavenger self-paced · 21445b09
      Michael Anthony Knyszek authored
      Currently the runtime background scavenger is paced externally,
      controlled by a collection of variables which together describe a line
      that we'd like to stay under.
      
      However, the line to stay under is computed as a function of the number
      of free and unscavenged huge pages in the heap at the end of the last
      GC. Aside from this number being inaccurate (which is still acceptable),
      the scavenging system also makes an order-of-magnitude assumption as to
      how expensive scavenging a single page actually is.
      
      This change simplifies the scavenger in preparation for making it
      operate on bitmaps. It makes it so that the scavenger paces itself, by
      measuring the amount of time it takes to scavenge a single page. The
      scavenging methods on mheap already avoid breaking huge pages, so if we
      scavenge a real huge page, then we'll have paced correctly, otherwise
      we'll sleep for longer to avoid using more than scavengePercent wall
      clock time.
      
      Unfortunately, all this involves measuring time, which is quite tricky.
      Currently we don't directly account for long process sleeps or OS-level
      context switches (which is quite difficult to do in general), but we do
      account for Go scheduler overhead and variations in it by maintaining an
      EWMA of the ratio of time spent scavenging to the time spent sleeping.
      This ratio, as well as the sleep time, are bounded in order to deal with
      the aforementioned OS-related anomalies.
      
      Updates #35112.
      
      Change-Id: Ieca8b088fdfca2bebb06bcde25ef14a42fd5216b
      Reviewed-on: https://go-review.googlesource.com/c/go/+/201763
      Run-TryBot: Michael Knyszek <mknyszek@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      21445b09
    • Brian Kessler's avatar
      math/cmplx: handle special cases · 68dce429
      Brian Kessler authored
      Implement special case handling and testing to ensure
      conformance with the C99 standard annex G.6 Complex arithmetic.
      
      Fixes #29320
      
      Change-Id: Ieb0527191dd7fdea5b1aecb42b9e23aae3f74260
      Reviewed-on: https://go-review.googlesource.com/c/go/+/169501
      Run-TryBot: Brian Kessler <brian.m.kessler@gmail.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarRobert Griesemer <gri@golang.org>
      68dce429
    • Cherry Zhang's avatar
      cmd/internal/obj/ppc64: handle MOVDU for SP delta · 3c47eada
      Cherry Zhang authored
      If a MOVDU instruction is used with an offset of SP, the
      instruction changes SP therefore needs an SP delta, which is used
      for generating the PC-SP table for stack unwinding. MOVDU is
      frequently used for allocating the frame and saving the LR in the
      same instruction, so this is particularly useful.
      
      Change-Id: Icb63eb55aa01c3dc350ac4e4cff6371f4c3c5867
      Reviewed-on: https://go-review.googlesource.com/c/go/+/205279
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      3c47eada
    • Cherry Zhang's avatar
      cmd/compile, cmd/internal/obj/ppc64: mark unsafe points · ceca99bd
      Cherry Zhang authored
      We'll use CTR as a scratch register for call injection. Mark code
      sequences that use CTR as unsafe for async preemption. Currently
      it is only used in LoweredZero and LoweredMove. It is unfortunate
      that they are nonpreemptible. But I think it is still better than
      using LR for call injection and marking all leaf functions
      nonpreemptible.
      
      Also mark the prologue of large frame functions nonpreemptible,
      as we write below SP.
      
      Change-Id: I05a75431499f3f4b2f23651a7b17f7fcf2afbe06
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203823
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      ceca99bd
    • Cherry Zhang's avatar
      cmd/compile, cmd/internal/obj/ppc64: use LR for indirect calls · 24e549a7
      Cherry Zhang authored
      On PPC64, indirect calls can be made through LR or CTR. Currently
      both are used. This CL changes it to always use LR.
      
      For async preemption, to return from the injected call, we need
      an indirect jump back to the PC we preeempted. This jump can be
      made through LR or CTR. So we'll have to clobber either LR or CTR.
      Currently, LR is used more frequently. In particular, for a leaf
      function, LR is live throughout the function. We don't want to
      make leaf functions nonpreemptible. So we choose CTR for the call
      injection. For code sequences that use CTR, if it is ok to use
      another register, change it to.
      
      Plus, it is a call so it will clobber LR anyway. It doesn't need
      to also clobber CTR (even without preemption).
      
      Change-Id: I07bd0e93b94a1a3aa2be2cd465801136165d8ab8
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203822
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      24e549a7
    • Cherry Zhang's avatar
      cmd/compile: mark unsafe points for MIPS and MIPS64 · a96cfa75
      Cherry Zhang authored
      Mark atomic LL/SC loops as unsafe for async preemption, as they
      use REGTMP.
      
      Change-Id: I5be7f93ad3ee337049ec7c3efd6fdc30eef87d97
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203719
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      a96cfa75
    • Cherry Zhang's avatar
      cmd/internal/obj/mips: mark unsafe points · 69dcdbd2
      Cherry Zhang authored
      For async preemption, we will be using REGTMP as a temporary
      register in injected call on MIPS, which will clobber it. So any
      code that uses REGTMP is not safe for async preemption.
      
      In the assembler backend, we expand a Prog to multiple machine
      instructions and use REGTMP as a temporary register if necessary.
      These need to be marked unsafe. In fact, most of the
      multi-instruction Progs use REGTMP, so we mark all of them,
      except ones that are whitelisted.
      
      Change-Id: Ic00ae5589683c2c9525abdaee076d884df6b0d1e
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203718
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      69dcdbd2
    • Cherry Zhang's avatar
      runtime: add async preemption support on ARM64 · 1b0b9809
      Cherry Zhang authored
      This CL adds support of call injection and async preemption on
      ARM64.
      
      There seems no way to return from the injected call without
      clobbering *any* register. So we have to clobber one, which is
      chosen to be REGTMP. Previous CLs have marked code sequences
      that use REGTMP async-nonpreemtible.
      
      Change-Id: Ieca4e3ba5557adf3d0f5d923bce5f1769b58e30b
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203461
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      Reviewed-by: default avatarAustin Clements <austin@google.com>
      1b0b9809
    • Cherry Zhang's avatar
      cmd/internal/obj/arm64: mark unsafe points · 47360884
      Cherry Zhang authored
      For async preemption, we will be using REGTMP as a temporary
      register in injected call on ARM64, which will clobber it. So any
      code that uses REGTMP is not safe for async preemption.
      
      In the assembler backend, we expand a Prog to multiple machine
      instructions and use REGTMP as a temporary register if necessary.
      These need to be marked unsafe. In fact, most of the
      multi-instruction Progs use REGTMP, so we mark all of them,
      except ones that are whitelisted.
      
      Change-Id: I6e97805a13950e3b693fb606d77834940ac3722e
      Reviewed-on: https://go-review.googlesource.com/c/go/+/203460
      Run-TryBot: Cherry Zhang <cherryyz@google.com>
      TryBot-Result: Gobot Gobot <gobot@golang.org>
      Reviewed-by: default avatarKeith Randall <khr@golang.org>
      47360884