runtime: implement STW GC in terms of concurrent GC

Currently, STW GC works very differently from concurrent GC. The
largest differences in that in concurrent GC, all marking work is done
by background mark workers during the mark phase, while in STW GC, all
marking work is done by gchelper during the mark termination phase.

This is a consequence of the evolution of Go's GC from a STW GC by
incrementally moving work from STW mark termination into concurrent
mark. However, at this point, the STW code paths exist only as a
debugging mode. Having separate code paths for this increases the
maintenance burden and complexity of the garbage collector. At the
same time, these code paths aren't tested nearly as well, making it
far more likely that they will bit-rot.

This CL reverses the relationship between STW GC, by re-implementing
STW GC in terms of concurrent GC.

This builds on the new scheduled support for disabling user goroutine
scheduling. During sweep termination, it disables user scheduling, so
when the GC starts the world again for concurrent mark, it's really
only "concurrent" with itself.

There are several code paths that were specific to STW GC that are now
vestigial. We'll remove these in the follow-up CLs.

Updates #26903.

Change-Id: Ia3883d2fcf7ab1d89bdc9c8ee54bf9bffb32c096
Reviewed-on: https://go-review.googlesource.com/c/134780
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>

src/runtime/mgc.go

@@ -455,6 +455,12 @@ func (c *gcControllerState) startCycle() {
 		c.fractionalUtilizationGoal = 0
 	}
 
+	// In STW mode, we just want dedicated workers.
+	if debug.gcstoptheworld > 0 {
+		c.dedicatedMarkWorkersNeeded = int64(gomaxprocs)
+		c.fractionalUtilizationGoal = 0
+	}
+
 	// Clear per-P state
 	for _, p := range allp {
 		p.gcAssistTime = 0
@@ -1264,9 +1270,7 @@ func gcStart(trigger gcTrigger) {
 		traceGCStart()
 	}
 
-	if mode == gcBackgroundMode {
 	gcBgMarkStartWorkers()
-	}
 
 	gcResetMarkState()
 
@@ -1296,10 +1300,17 @@ func gcStart(trigger gcTrigger) {
 	clearpools()
 
 	work.cycles++
-	if mode == gcBackgroundMode { // Do as much work concurrently as possible
+
 	gcController.startCycle()
 	work.heapGoal = memstats.next_gc
 
+	// In STW mode, disable scheduling of user Gs. This may also
+	// disable scheduling of this goroutine, so it may block as
+	// soon as we start the world again.
+	if mode != gcBackgroundMode {
+		schedEnableUser(false)
+	}
+
 	// Enter concurrent mark phase and enable
 	// write barriers.
 	//
@@ -1340,21 +1351,14 @@ func gcStart(trigger gcTrigger) {
 	// Concurrent mark.
 	systemstack(func() {
 		now = startTheWorldWithSema(trace.enabled)
-		})
 		work.pauseNS += now - work.pauseStart
 		work.tMark = now
-	} else {
-		if trace.enabled {
-			// Switch to mark termination STW.
-			traceGCSTWDone()
-			traceGCSTWStart(0)
-		}
-		t := nanotime()
-		work.tMark, work.tMarkTerm = t, t
-		work.heapGoal = work.heap0
-
-		// Perform mark termination. This will restart the world.
-		gcMarkTermination(memstats.triggerRatio)
+	})
+	// In STW mode, we could block the instant systemstack
+	// returns, so don't do anything important here. Make sure we
+	// block rather than returning to user code.
+	if mode != gcBackgroundMode {
+		Gosched()
 	}
 
 	semrelease(&work.startSema)
@@ -1468,6 +1472,10 @@ top:
 	// world again.
 	semrelease(&work.markDoneSema)
 
+	// In STW mode, re-enable user goroutines. These will be
+	// queued to run after we start the world.
+	schedEnableUser(true)
+
 	// endCycle depends on all gcWork cache stats being flushed.
 	// The termination algorithm above ensured that up to
 	// allocations since the ragged barrier.