runtime: clear tiny alloc cache in mark term, not sweep term

The tiny alloc cache is maintained in a pointer from non-GC'd memory
(mcache) to heap memory and hence must be handled carefully.

Currently we clear the tiny alloc cache during sweep termination and,
if it is assigned to a non-nil value during concurrent marking, we
depend on a write barrier to keep the new value alive. However, while
the compiler currently always generates this write barrier, we're
treading on thin ice because write barriers may not happen for writes
to non-heap memory (e.g., typedmemmove). Without this lucky write
barrier, the GC may free a current tiny block while it's still
reachable by the tiny allocator, leading to later memory corruption.

Change this code so that, rather than depending on the write barrier,
we simply clear the tiny cache during mark termination when we're
clearing all of the other mcaches. If the current tiny block is
reachable from regular pointers, it will be retained; if it isn't
reachable from regular pointers, it may be freed, but that's okay
because there won't be any pointers in non-GC'd memory to it.

Change-Id: I8230980d8612c35c2997b9705641a1f9f865f879
Reviewed-on: https://go-review.googlesource.com/16962
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

src/runtime/mcache.go

@@ -8,14 +8,25 @@ import "unsafe"
 
 // Per-thread (in Go, per-P) cache for small objects.
 // No locking needed because it is per-thread (per-P).
+//
+// mcaches are allocated from non-GC'd memory, so any heap pointers
+// must be specially handled.
 type mcache struct {
 	// The following members are accessed on every malloc,
 	// so they are grouped here for better caching.
 	next_sample      int32   // trigger heap sample after allocating this many bytes
 	local_cachealloc uintptr // bytes allocated from cache since last lock of heap
 	local_scan       uintptr // bytes of scannable heap allocated
+
 	// Allocator cache for tiny objects w/o pointers.
 	// See "Tiny allocator" comment in malloc.go.
+
+	// tiny points to the beginning of the current tiny block, or
+	// nil if there is no current tiny block.
+	//
+	// tiny is a heap pointer. Since mcache is in non-GC'd memory,
+	// we handle it by clearing it in releaseAll during mark
+	// termination.
 	tiny             unsafe.Pointer
 	tinyoffset       uintptr
 	local_tinyallocs uintptr // number of tiny allocs not counted in other stats
@@ -127,4 +138,7 @@ func (c *mcache) releaseAll() {
 			c.alloc[i] = &emptymspan
 		}
 	}
+	// Clear tinyalloc pool.
+	c.tiny = nil
+	c.tinyoffset = 0
 }

src/runtime/mgc.go

@@ -1744,17 +1744,6 @@ func clearpools() {
 		sched.deferpool[i] = nil
 	}
 	unlock(&sched.deferlock)
-
-	for _, p := range &allp {
-		if p == nil {
-			break
-		}
-		// clear tinyalloc pool
-		if c := p.mcache; c != nil {
-			c.tiny = nil
-			c.tinyoffset = 0
-		}
-	}
 }
 
 // Timing