Fixes #11410

Change-Id: I9d01be3e01f74f4b12a516aba8a5d20d9b277ec6
Reviewed-on: https://go-review.googlesource.com/11539Reviewed-by: Russ Cox <rsc@golang.org>

Update #11443

Change-Id: Icb7ea291a837dcf2799a791a2ba780fd2a5e712b
Reviewed-on: https://go-review.googlesource.com/11721Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>

Should fix arm64 build failure.

Change-Id: Ib35e4a69a1082e461a1eddf3265544a1d2ff98e5
Reviewed-on: https://go-review.googlesource.com/11710Reviewed-by: David Crawshaw <crawshaw@golang.org>

Fixes #8836.

Change-Id: Idda9f4a987e03b3bdf5e8fdb984fe56d6f84aa59
Reviewed-on: https://go-review.googlesource.com/11672Reviewed-by: David Chase <drchase@google.com>

Change-Id: I08de33aacb3fc932722286d69b1dd70ffe787c89
Reviewed-on: https://go-review.googlesource.com/11697Reviewed-by: Russ Cox <rsc@golang.org>

The old code was recording the current table output offset,
so the table from the next function would be used instead of
the runtime realizing that there was no table at all.

Add debug constant in runtime to check this for every function
at startup. It's too expensive to do that by default, but we can
do the last five functions. The end of the table is usually where
the C symbols end up, so that's where the problems typically are.

Fixes #10747.
Fixes #11396.

Change-Id: I13592e78017969fc22979fa902e19e1b151d41b1
Reviewed-on: https://go-review.googlesource.com/11657Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>

Currently we fail to reset the live heap accounting state before the
checkmark mark and before the gctrace=2 extra mark. As a result, if
either are enabled, at the end of GC it thinks there are 0 bytes of
live heap, which causes the GC controller to initiate a new GC
immediately, regardless of the true heap size.

Fix this by factoring this state reset into a function and calling it
before all three possible marks.

This function should be merged with gcResetGState, but doing so
requires some additional cleanup, so it will wait for after the
freeze. Filed #11427 for this cleanup.

Fixes #10492.

Change-Id: Ibe46348916fc8368fac6f086e142815c970a6f4d
Reviewed-on: https://go-review.googlesource.com/11561Reviewed-by: Russ Cox <rsc@golang.org>

Memory for stacks is manually managed by the runtime and, currently
(with one exception) we free stack spans immediately when the last
stack on a span is freed. However, the garbage collector assumes that
spans can never transition from non-free to free during scan or mark.
This disagreement makes it possible for the garbage collector to mark
uninitialized objects and is blocking us from re-enabling the bad
pointer test in the garbage collector (issue #9880).

For example, the following sequence will result in marking an
uninitialized object:

1. scanobject loads a pointer slot out of the object it's scanning.
   This happens to be one of the special pointers from the heap into a
   stack. Call the pointer p and suppose it points into X's stack.

2. X, running on another thread, grows its stack and frees its old
   stack.

3. The old stack happens to be large or was the last stack in its
   span, so X frees this span, setting it to state _MSpanFree.

4. The span gets reused as a heap span.

5. scanobject calls heapBitsForObject, which loads the span containing
   p, which is now in state _MSpanInUse, but doesn't necessarily have
   an object at p. The not-object at p gets marked, and at this point
   all sorts of things can go wrong.

We already have a partial solution to this. When shrinking a stack, we
put the old stack on a queue to be freed at the end of garbage
collection. This was done to address exactly this problem, but wasn't
a complete solution.

This commit generalizes this solution to both shrinking and growing
stacks. For stacks that fit in the stack pool, we simply don't free
the span, even if its reference count reaches zero. It's fine to reuse
the span for other stacks, and this enables that. At the end of GC, we
sweep for cached stack spans with a zero reference count and free
them. For larger stacks, we simply queue the stack span to be freed at
the end of GC. Ideally, we would reuse these large stack spans the way
we can small stack spans, but that's a more invasive change that will
have to wait until after the freeze.

Fixes #11267.

Change-Id: Ib7f2c5da4845cc0268e8dc098b08465116972a71
Reviewed-on: https://go-review.googlesource.com/11502Reviewed-by: Russ Cox <rsc@golang.org>

We don't use this state. _GCoff means we're sweeping in the
background. This makes it clear in the next commit that _GCoff and
only _GCoff means sweeping.

Change-Id: I416324a829ba0be3794a6cf3cf1655114cb6e47c
Reviewed-on: https://go-review.googlesource.com/11501Reviewed-by: Rick Hudson <rlh@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

Port of https://golang.org/cl/154360045 to Git.
Original author is Xia Bin <snyh@snyh.org> (already a contributor).

Fixes #8945.

Change-Id: I28bcaf3348794202ca59fbc3466bd7b9670030e4
Reviewed-on: https://go-review.googlesource.com/11658Reviewed-by: Ian Lance Taylor <iant@golang.org>

Currently the runtime fails to clear a G's stack barriers in gfput if
the G's stack allocation is _FixedStack bytes. This causes the runtime
to panic if the following sequence of events happens:

1) The runtime installs stack barriers on a G.

2) The G exits by calling runtime.Goexit. Since this does not
   necessarily return through the stack barriers installed on the G,
   there may still be untriggered stack barriers left on the G's stack
   in recorded in g.stkbar.

3) The runtime calls gfput to add the exiting G to the free pool. If
   the G's stack allocation is _FixedStack bytes, we fail to clear
   g.stkbar.

4) A new G starts and allocates the G that was just added to the free
   pool.

5) The new G begins to execute and overwrites the stack slots that had
   stack barriers in them.

6) The garbage collector enters mark termination, attempts to remove
   stack barriers from the new G, and finds that they've been
   overwritten.

Fix this by clearing the stack barriers in gfput in the case where it
reuses the stack.

Fixes #11256.

Change-Id: I377c44258900e6bcc2d4b3451845814a8eeb2bcf
Reviewed-on: https://go-review.googlesource.com/11461Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Reviewed-by: Russ Cox <rsc@golang.org>

Updated the protobuf documentation URL (code.google.com deprecated)
to avoid a redirect.

Change-Id: I134f6e4a2bf2bba699942883bf6347bc61700bcb
Reviewed-on: https://go-review.googlesource.com/11634Reviewed-by: Russ Cox <rsc@golang.org>

Also improve the documentation. A prior fix in this release
changed the properties for empty strings and slices, incorrectly.
Previous behavior is now restored and better documented.

Add lots of tests.

The behavior is that when using a string-like format (%s %q %x %X)
a byte slice is equivalent to a string, and printed as a unit. The padding
applies to the entire object. (The space and sharp flags apply
elementwise.)

Fixes #11422.
Fixes #10430.

Change-Id: I758f0521caf71630437e43990ec6d6c9a92655e3
Reviewed-on: https://go-review.googlesource.com/11600Reviewed-by: Russ Cox <rsc@golang.org>

Change-Id: Id2cfa63d4c749503f729097654d7cbd2b252f192
Reviewed-on: https://go-review.googlesource.com/11660Reviewed-by: Andrew Gerrand <adg@golang.org>

Change-Id: I4befb21d0811819ce0a5721421a2f6df7a9b62fa
Reviewed-on: https://go-review.googlesource.com/11605Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>

TestTransportAndServerSharedBodyRace got flaky after
issue #9662 was fixed by https://golang.org/cl/11412, which made
servers hang up on clients when a Handler stopped reading its body
early.

This test was affected by a race between the the two goroutines in the
test both only reading part of the request, which was an unnecessary
detail for what the test was trying to test (concurrent Read/Close
races on an *http.body)

Also remove an unused remnant from an old test from which this one was
derived. And make the test not deadlock when it fails. (which was why
the test was showing up as 2m timeouts on the dashboard)

Fixes #11418

Change-Id: Ic83d18aef7e09a9cd56ac15e22ebed75713026cb
Reviewed-on: https://go-review.googlesource.com/11610
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>

Fixes #11417

Change-Id: Iacea829a48b39df0a4f751b06b19e918fbb713d0
Reviewed-on: https://go-review.googlesource.com/11604Reviewed-by: Rob Pike <r@golang.org>

It looks like the test for whether symbols contain subsymbols is wrong.
In particular, symbols in C libraries are mistakenly considered container
symbols.

Fix the test so only symbols which actually have a subsymbol
are excluded from the symtab.  When linking cgo programs the list
of containers is small, something like:

container _/home/khr/sandbox/symtab/misc/cgo/test(.text)<74>
container _/home/khr/sandbox/symtab/misc/cgo/test/issue8828(.text)<75>
container _/home/khr/sandbox/symtab/misc/cgo/test/issue9026(.text)<76>
container runtime/cgo(.text)<77>

I'm not sure this is the right fix.  In particular I can't reproduce
the original problem.  Anyone have a repro they can try and see if
this fix works?

Fixes #10747
Fixes #11396

Change-Id: Id8b016389d33348b4a791fdcba0f9db8ae71ebf3
Reviewed-on: https://go-review.googlesource.com/11652Reviewed-by: Russ Cox <rsc@golang.org>

Invalid UTF-8 triggers an error when marshaling but, previously, not
when unmarshaling. This means that ASN.1 structures were not
round-tripping.

This change makes invalid UTF-8 in a string marked as UTF-8 to be an
error when Unmarshaling.

Fixes #11126.

Change-Id: Ic37be84d21dc5c03983525e244d955a8b1e1ff14
Reviewed-on: https://go-review.googlesource.com/11056
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

The time package does normalisation of times: for example day zero is
converted to the last day of the previous month and the 31st of February
is moved into March etc. This makes the ASN.1 parsing a little
worryingly lax.

This change causes the parser to reserialise parsed times to ensure that
they round-trip correctly and thus were not normalised.

Fixes #11134.

Change-Id: I3988bb95153a7b33d64ab861fbe51b1a34a359e9
Reviewed-on: https://go-review.googlesource.com/11094
Run-TryBot: Adam Langley <agl@golang.org>
Reviewed-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

Stack can move during callback, so libcall struct cannot be stored on stack.
asmstdcall updates return values and errno in libcall struct parameter, but
these could be at different location when callback returns.
Store these in m, so they are not affected by GC.

Fixes #10406

Change-Id: Id01c9d2b4b44530494e6d9e9e1c875261ce477cd
Reviewed-on: https://go-review.googlesource.com/10370Reviewed-by: Russ Cox <rsc@golang.org>

git sets read-only flag on all its repo files on Windows.
os.Remove cannot delete these files.

Fixes windows build

Change-Id: Icaf72470456b88a1c26295caecd4e0d3dc22a1b6
Reviewed-on: https://go-review.googlesource.com/11602Reviewed-by: Alex Brainman <alex.brainman@gmail.com>

Change-Id: I67947e0e3189093e830120941ee49f9f32086f0e
Reviewed-on: https://go-review.googlesource.com/11615Reviewed-by: Rob Pike <r@golang.org>

The race occurs rarely, but by putting some delays and more reads/writes
of prePendingDial/postPendingDial in the handlePendingDial function I
could reproduce it.

Fixes #11136

Change-Id: I8da9e66c88fbda049eaaaaffa2717264ef327768
Reviewed-on: https://go-review.googlesource.com/11250Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>

AFAIK, the documentation does not explicitly state whether
variables can store a callable entity or not. I believe the
current implementation in text/template assumes they cannot
though. The call builtin function is supposed to be used for
this purpose.

Template "{{0|$}}" should generate an error at runtime,
instead of a panic.

Similarly, template "{{0|(nil)}}" should not generate
a panic.

This CL aborts the sanitization process for a given pipeline
when no identifier can be derived from the selected node.
It happens with malformed pipelines.

We now have the following errors:

{{ 0 | $ }}
template: foo:1:10: executing "foo" at <$>: can't give argument to non-function $

{{ 0 | (nil) }}
template: foo:1:11: executing "foo" at <nil>: nil is not a command

Fixes #11118
Fixes #11356

Change-Id: Idae52f806849f4c9ab7aca1b4bb4b59a74723d0e
Reviewed-on: https://go-review.googlesource.com/10823Reviewed-by: Rob Pike <r@golang.org>

Partial revert of cl/10284 to get -buildmode=c-archive working for
darwin/arm.

Manually tested with iostest.bash while builder is offline.

Change-Id: I98e4e209765666e320e680e11151fce59e2afde9
Reviewed-on: https://go-review.googlesource.com/11306Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: David Crawshaw <crawshaw@golang.org>

Fixes #10952

Change-Id: I56ab6a806bd3741cffd9d2a53929a6d043626a26
Reviewed-on: https://go-review.googlesource.com/10693Reviewed-by: Russ Cox <rsc@golang.org>

Change createCmd, downloadCmd, tagSyncCmd, tagSyncDefault to allow
multiple commands.

When using the vendoring experiment, fetch git submodules in `go get`,
and update them in `go get -u`.

This is a reincarnation of https://codereview.appspot.com/142180043.

For #7764.

Change-Id: I8248efb851130620ef762a765ab8716af430572a
Reviewed-on: https://go-review.googlesource.com/9815Reviewed-by: Russ Cox <rsc@golang.org>

cmd/go sometimes returns relative path in the error message
(see shortPath function). Account for that during
TestFileLineInErrorMessages.

Fixes #11355

Change-Id: Ica79359eab48d669d307449fdd458764895fab2c
Reviewed-on: https://go-review.googlesource.com/11475Reviewed-by: Russ Cox <rsc@golang.org>

- Let runOutput return the error message
- When `git config ...` returns empty buffer, it means the config key is
  correct, but there is no corresponding value.
- Return the correct error when the url of remote origin is not found.
- Update error message

Fixes: #10922

Change-Id: I3f8880f6717a4f079b840d1249174378d36bca1b
Reviewed-on: https://go-review.googlesource.com/10475Reviewed-by: Russ Cox <rsc@golang.org>

Linux allows to have a peer IP address on IP interface over ethernet
link encapsulation, though it only installs a static route with the peer
address as an on-link nexthop.

Fixes #11338.

Change-Id: Ie2583737e4c7cec39baabb89dd732463d3f10a61
Reviewed-on: https://go-review.googlesource.com/11352Reviewed-by: Russ Cox <rsc@golang.org>

Also add a couple more errors, such as modulo with a zero divisor.

Change-Id: If24c95477f7ae86cf4aef5b3460e9ec249ea5ae2
Reviewed-on: https://go-review.googlesource.com/11535Reviewed-by: Russ Cox <rsc@golang.org>

Update #11426

Change-Id: I7abc4ed2241a7a3af6d57c934786f36de4f97b77
Reviewed-on: https://go-review.googlesource.com/11592
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Fixes the build from https://golang.org/cl/4117 (sync: simplify WaitGroup)

Change-Id: Icc2a7ba8acea26fd187d52cf1901bfebf8573f93
Reviewed-on: https://go-review.googlesource.com/11591Reviewed-by: Austin Clements <austin@google.com>

Currently, to write out the bitmap of a slice of a type with a GCprog,
we construct a new GCprog that executes the underlying type's GCprog
to write out the bitmap once and then repeats those bits n more times.
This results in n+1 repetitions of the bitmap, which is one more
repetition than it should be. This corrupts the bitmap of the heap
following the slice and may write past the mapped bitmap memory and
segfault.

Fix this by repeating the bitmap only n-1 more times.

Fixes #11430.

Change-Id: Ic24854363bffc5a755b66f257339f9309ada3aa5
Reviewed-on: https://go-review.googlesource.com/11570
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Change-Id: Ib80829e7cbfb319549a224dc18931ca884c8296a
Reviewed-on: https://go-review.googlesource.com/11532Reviewed-by: Russ Cox <rsc@golang.org>

Update #11426

Change-Id: If406d2efcc81965825a63c76f5448d544ba2a740
Reviewed-on: https://go-review.googlesource.com/11590Reviewed-by: Austin Clements <austin@google.com>

These tests pass after cl/11417.

Change-Id: Id98088c52e564208ce432e9717eddd672c42c66d
Reviewed-on: https://go-review.googlesource.com/11551Reviewed-by: Russ Cox <rsc@golang.org>

A comment in waitgroup.go describes the following scenario
as the reason to have dynamically created semaphores:

// G1: Add(1)
// G1: go G2()
// G1: Wait() // Context switch after Unlock() and before Semacquire().
// G2: Done() // Release semaphore: sema == 1, waiters == 0. G1 doesn't run yet.
// G3: Wait() // Finds counter == 0, waiters == 0, doesn't block.
// G3: Add(1) // Makes counter == 1, waiters == 0.
// G3: go G4()
// G3: Wait() // G1 still hasn't run, G3 finds sema == 1, unblocked! Bug.

However, the scenario is incorrect:
G3: Add(1) happens concurrently with G1: Wait(),
and so there is no reasonable behavior of the program
(G1: Wait() may or may not wait for G3: Add(1) which
can't be the intended behavior).

With this conclusion we can:
1. Remove dynamic allocation of semaphores.
2. Remove the mutex entirely and instead pack counter and waiters
   into single uint64.

This makes the logic significantly simpler, both Add and Wait
do only a single atomic RMW to update the state.

benchmark                            old ns/op     new ns/op     delta
BenchmarkWaitGroupUncontended        30.6          32.7          +6.86%
BenchmarkWaitGroupActuallyWait       722           595           -17.59%
BenchmarkWaitGroupActuallyWait-2     396           319           -19.44%
BenchmarkWaitGroupActuallyWait-4     224           183           -18.30%
BenchmarkWaitGroupActuallyWait-8     134           106           -20.90%

benchmark                          old allocs     new allocs     delta
BenchmarkWaitGroupActuallyWait     2              1              -50.00%

benchmark                          old bytes     new bytes     delta
BenchmarkWaitGroupActuallyWait     48            16            -66.67%

Change-Id: I28911f3243aa16544e99ac8f1f5af31944c7ea3a
Reviewed-on: https://go-review.googlesource.com/4117
Run-TryBot: Dmitry Vyukov <dvyukov@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

Fixes #10572

Change-Id: I764f3c226cf98ff39d9e553e4613d0ee108ef766
Reviewed-on: https://go-review.googlesource.com/9311Reviewed-by: Russ Cox <rsc@golang.org>