Moves sweep phase out of stoptheworld by adding
background sweeper goroutine and lazy on-demand sweeping.

It turned out to be somewhat trickier than I expected,
because there is no point in time when we know size of live heap
nor consistent number of mallocs and frees.
So everything related to next_gc, mprof, memstats, etc becomes trickier.

At the end of GC next_gc is conservatively set to heap_alloc*GOGC,
which is much larger than real value. But after every sweep
next_gc is decremented by freed*GOGC. So when everything is swept
next_gc becomes what it should be.

For mprof I had to introduce 3-generation scheme (allocs, revent_allocs, prev_allocs),
because by the end of GC we know number of frees for the *previous* GC.

Significant caution is required to not cross yet-unknown real value of next_gc.
This is achieved by 2 means:
1. Whenever I allocate a span from MCentral, I sweep a span in that MCentral.
2. Whenever I allocate N pages from MHeap, I sweep until at least N pages are
returned to heap.
This provides quite strong guarantees that heap does not grow when it should now.

http-1
allocated                    7036         7033      -0.04%
allocs                         60           60      +0.00%
cputime                     51050        46700      -8.52%
gc-pause-one             34060569      1777993     -94.78%
gc-pause-total               2554          133     -94.79%
latency-50                 178448       170926      -4.22%
latency-95                 284350       198294     -30.26%
latency-99                 345191       220652     -36.08%
rss                     101564416    101007360      -0.55%
sys-gc                    6606832      6541296      -0.99%
sys-heap                 88801280     87752704      -1.18%
sys-other                 7334208      7405928      +0.98%
sys-stack                  524288       524288      +0.00%
sys-total               103266608    102224216      -1.01%
time                        50339        46533      -7.56%
virtual-mem             292990976    293728256      +0.25%

garbage-1
allocated                 2983818      2990889      +0.24%
allocs                      62880        62902      +0.03%
cputime                  16480000     16190000      -1.76%
gc-pause-one            828462467    487875135     -41.11%
gc-pause-total            4142312      2439375     -41.11%
rss                    1151709184   1153712128      +0.17%
sys-gc                   66068352     66068352      +0.00%
sys-heap               1039728640   1039728640      +0.00%
sys-other                37776064     40770176      +7.93%
sys-stack                 8781824      8781824      +0.00%
sys-total              1152354880   1155348992      +0.26%
time                     16496998     16199876      -1.80%
virtual-mem            1409564672   1402281984      -0.52%

LGTM=rsc
R=golang-codereviews, sameer, rsc, iant, jeremyjackins, gobot
CC=golang-codereviews, khr
https://golang.org/cl/46430043

$ go test -cpu=1,1,1,1,1,1,1,1,1 encoding/json
--- FAIL: TestIndentBig (0.00 seconds)
        scanner_test.go:131: Indent(jsonBig) did not get bigger
On 4-th run initBig generates an empty array.

LGTM=bradfitz
R=golang-codereviews, bradfitz
CC=golang-codereviews
https://golang.org/cl/49930051

ConstantTimeCompare has always been documented to take equal length
slices but perhaps this is too subtle, even for 'subtle'.

Fixes #7304.

LGTM=hanwen, bradfitz
R=golang-codereviews, hanwen, bradfitz
CC=golang-codereviews
https://golang.org/cl/62190043

LGTM=bradfitz
R=golang-codereviews, bradfitz
CC=golang-codereviews
https://golang.org/cl/60580046

Allow clients to check for timeouts without relying on error substring
matching.

Fixes #6185.

LGTM=bradfitz
R=golang-codereviews, bradfitz
CC=golang-codereviews
https://golang.org/cl/55470048

Fixes #7104.

LGTM=iant
R=golang-dev, iant
CC=golang-codereviews
https://golang.org/cl/61480049

Generated by addca.

R=gobot
CC=golang-codereviews
https://golang.org/cl/53410046

Fixes #7102.

LGTM=adonovan
R=adonovan
CC=golang-codereviews
https://golang.org/cl/56770045

LGTM=bradfitz
R=golang-codereviews, bradfitz
CC=golang-codereviews
https://golang.org/cl/54830045

Second part of the solaris/amd64 linker changes: relocation and symbol table.

LGTM=iant
R=golang-codereviews, iant
CC=golang-codereviews
https://golang.org/cl/61330043

Fixes #7305.

LGTM=adonovan
R=bradfitz, adonovan
CC=golang-codereviews
https://golang.org/cl/58950045

The UNDEF instruction was listed in the instruction data as having the next instruction in the stream as its successor. This confused the optimizer into adding a load where it wasn't needed, in turn confusing the liveness analysis pass for GC bitmaps into thinking that the variable was live.

Fixes #7229.

LGTM=iant, rsc
R=golang-codereviews, bradfitz, iant, dave, rsc
CC=golang-codereviews
https://golang.org/cl/56910045

Prevents a ton of garbage. (Noticed this when writing large
Camlistore zip archives to Amazon Glacier)

Note that the Closer part of the io.WriteCloser is never given
to users. It's an internal detail of the package.

benchmark                         old ns/op     new ns/op     delta
BenchmarkCompressedZipGarbage     42884123      40732373      -5.02%

benchmark                         old allocs     new allocs     delta
BenchmarkCompressedZipGarbage     204            149            -26.96%

benchmark                         old bytes     new bytes     delta
BenchmarkCompressedZipGarbage     4397576       66744         -98.48%

LGTM=adg, rsc
R=adg, rsc
CC=golang-codereviews
https://golang.org/cl/54300053

There is frequently a thread hanging on GQCS,
currently it skews profiles towards netpoll,
but it is not bad and is not consuming any resources.

R=alex.brainman
CC=golang-codereviews
https://golang.org/cl/61560043

LGTM=rsc
R=rsc
CC=golang-codereviews
https://golang.org/cl/61410046

Generated by addca.

R=gobot
CC=golang-codereviews
https://golang.org/cl/61410050

LGTM=dsymonds
R=dsymonds
CC=golang-codereviews
https://golang.org/cl/61070047

Fixes #7292.

LGTM=dsymonds
R=dsymonds
CC=golang-codereviews
https://golang.org/cl/61650046

It was deleted from Vimeo. I will attempt to get a new version online.

Fixes #7297.

LGTM=r
R=r
CC=golang-codereviews
https://golang.org/cl/61740043

At present, when a package identifier is used outside of a selector expression, gc gives the error "use of package %S outside selector". However, in the selector expression x.f, the spec defines f as the selector. This change makes the error clearer.

Fixes #7133.

LGTM=rsc
R=golang-codereviews, rsc
CC=golang-codereviews
https://golang.org/cl/50060047

Generated by addca.

R=gobot
CC=golang-codereviews
https://golang.org/cl/55870046

This got deleted mistakenly during the content re-org.

Fixes #7275.

LGTM=dvyukov
R=golang-codereviews, minux.ma, dvyukov
CC=golang-codereviews
https://golang.org/cl/60850043

(per suggestion by Doug McIlroy)

LGTM=r
R=r
CC=golang-codereviews
https://golang.org/cl/50580046

Fixes #7286.

LGTM=rsc
R=rsc
CC=golang-codereviews
https://golang.org/cl/61410044

LGTM=gri
R=golang-codereviews, gri
CC=golang-codereviews
https://golang.org/cl/61610043

R=r
CC=golang-codereviews
https://golang.org/cl/61650044

mp->mcache can be concurrently modified by runtime·helpgc.
In such case sigprof can remember mcache=nil, then helpgc sets it to non-nil,
then sigprof restores it back to nil, GC crashes with nil mcache.

R=rsc
CC=golang-codereviews
https://golang.org/cl/58860044

R=golang-codereviews, bradfitz
CC=golang-codereviews
https://golang.org/cl/60760048

Currently this leads to a significant skew towards 'etext' entry,
since all idle threads are profiled every tick.
Before:
Total: 66608 samples
   63188  94.9%  94.9%    63188  94.9% etext
     278   0.4%  95.3%      278   0.4% sweepspan
     216   0.3%  95.6%      448   0.7% runtime.mallocgc
     122   0.2%  95.8%      122   0.2% scanblock
     113   0.2%  96.0%      113   0.2% net/textproto.canonicalMIMEHeaderKey
After:
Total: 8008 samples
    3949  49.3%  49.3%     3949  49.3% etext
     231   2.9%  52.2%      231   2.9% scanblock
     211   2.6%  54.8%      211   2.6% runtime.cas64
     182   2.3%  57.1%      408   5.1% runtime.mallocgc
     178   2.2%  59.3%      178   2.2% runtime.atomicload64

LGTM=alex.brainman
R=golang-codereviews, alex.brainman
CC=golang-codereviews
https://golang.org/cl/61250043

LGTM=bradfitz
R=golang-codereviews, bradfitz, alex.brainman
CC=golang-codereviews
https://golang.org/cl/61260044

Fixes #7293.

Update #7261

The bsd ld(1) does not understand $ORIGIN and has restrictions on using -rpath when using clang(1), the default compiler on darwin.

LGTM=iant
R=iant
CC=golang-codereviews
https://golang.org/cl/58480045

Fixes #7260.

Fix three broken tests in test.bash

The test for issue 4568 was confused by go $ACTION . producing a package root of "", avoiding this mode fixes the test but weakens the test.

The test for issue 4773 was broken on linux because math/Rand would fail to resolve as a package causing the test for duplicates to be skipped.

Finally, the last breakage was a small change in the error message.

Also, add test for foldDup.

LGTM=iant
R=iant, rsc
CC=golang-codereviews
https://golang.org/cl/61070044

LGTM=adg
R=adg
CC=golang-codereviews
https://golang.org/cl/60530049

rsc suggested that we split the whole linker changes into three parts.
This is the first one, mostly dealing with adding Hsolaris.

LGTM=iant
R=golang-codereviews, iant, dave
CC=golang-codereviews
https://golang.org/cl/54210050

LGTM=bradfitz
R=dave, bradfitz
CC=golang-codereviews
https://golang.org/cl/61310043

LGTM=iant
R=golang-codereviews, iant
CC=golang-codereviews
https://golang.org/cl/54300054

This changes makes sgen and clearfat use unaligned instructions for
the trailing bytes, like the runtime memmove does, resulting in faster
code when manipulating types whose size is not a multiple of 8.

LGTM=khr
R=khr, iant, rsc
CC=golang-codereviews
https://golang.org/cl/51740044

LGTM=iant
R=iant
CC=golang-codereviews
https://golang.org/cl/60590047

LGTM=iant
R=golang-codereviews, dave, iant
CC=golang-codereviews
https://golang.org/cl/60740044

LGTM=minux.ma, dvyukov
R=golang-codereviews, minux.ma, dvyukov
CC=golang-codereviews
https://golang.org/cl/60900043