Change control flow to probe with N=1. This calls benchFunc
the same number of times as the old implementation in the
absence of subbenchmarks.

To be compatible with existing tools, benchmarking only
prints a line for "leaf" benchmarks. This means, though, that
the name of a benchmark can only be printed after the first
iteration.

Issue #14863

Change-Id: Ic7b9b89b058f8ebb5287755f24f9e47df8c9537c
Reviewed-on: https://go-review.googlesource.com/21043
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>

Also added go:generate directive.

Change-Id: Ib1f0eddc75e3c47a4d904786a29b964a35b18456
Reviewed-on: https://go-review.googlesource.com/21042Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>

This change removes a lot of dead code. Some of the code has never been
used, not even when it was first commited. The rest shouldn't have
survived refactors.

This change doesn't remove unused routines helpful for debugging, nor
does it remove code that's used in commented out blocks of code that are
only unused temporarily. Furthermore, unused constants weren't removed
when they were part of a set of constants from specifications.

One noteworthy omission from this CL are about 1000 lines of unused code
in cmd/fix, 700 lines of which are the typechecker, which hasn't been
used ever since the pre-Go 1 fixes have been removed. I wasn't sure if
this code should stick around for future uses of cmd/fix or be culled as
well.

Change-Id: Ib714bc7e487edc11ad23ba1c3222d1fd02e4a549
Reviewed-on: https://go-review.googlesource.com/20926Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

Store already padded keys instead of storing key and padding it during
Reset and Sum. This simplifies code and makes Reset-Write-Sum sequences
faster, which helps /x/crypto/pbkdf2.

HMAC benchmark:

benchmark                    old ns/op     new ns/op     delta
BenchmarkHMACSHA256_1K-4     7669          7613          -0.73%
BenchmarkHMACSHA256_32-4     1880          1737          -7.61%

benchmark                    old MB/s     new MB/s     speedup
BenchmarkHMACSHA256_1K-4     133.52       134.50       1.01x
BenchmarkHMACSHA256_32-4     17.02        18.41        1.08x

PBKDF2 benchmark:

benchmark                       old ns/op     new ns/op     delta
BenchmarkPBKDF2HMACSHA256-4     1943196       1807699       -6.97%

Change-Id: I6697028370c226715ab477b0844951a83eb3488c
Reviewed-on: https://go-review.googlesource.com/21024
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Adam Langley <agl@golang.org>

The Lookup method provides a way to extract a tag value, while
determining whether the tag key exists in the struct field's tag.

Fixes #14883

Change-Id: I7460cb68f0ca1aaa025935050b9e182efcb64db3
Reviewed-on: https://go-review.googlesource.com/20864Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

Fixes #14937.

Change-Id: Iea11a32230d44d5a43f8aec812d25f13bce85895
Reviewed-on: https://go-review.googlesource.com/21038Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

Fixes #6492.

Change-Id: Ibc633c43a6d134bb140addb59780a5758b35a5c5
Reviewed-on: https://go-review.googlesource.com/21057
Run-TryBot: Caleb Spare <cespare@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Ther darwin/arm{,64} exec wrapper now limits the number of concurrent
executions to 1, so remove the higher level parallel task limit from
the Go command.

Change-Id: Id84f65c3908305bde0452b3c8db6df8c5a8881bb
Reviewed-on: https://go-review.googlesource.com/21100Reviewed-by: David Crawshaw <crawshaw@golang.org>

I failed to rebase (and re-test) CL 21102 before submit, which meant
that two extra tests sneaked into testcarchive that still referenced
runtime.GOOS and runtime.GOARCH.

Convert the new tests.

While we're here, make sure pending tasks are flushed before running
the host tests. If not, the "##### misc/cgo/testcarchive" banner
and "PASS" won't show up in the all.bash output.

Change-Id: I41fc4ec9515f9a193fa052f7c31fac452153c897
Reviewed-on: https://go-review.googlesource.com/21106
Run-TryBot: Elias Naur <elias.naur@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

Create a byte encoding designed for static Go names.

It is intended to be a compact representation of a name
and optional tag data that can be turned into a Go string
without allocating, and describes whether or not it is
exported without unicode table.

The encoding is described in reflect/type.go:

// The first byte is a bit field containing:
//
//	1<<0 the name is exported
//	1<<1 tag data follows the name
//	1<<2 pkgPath *string follow the name and tag
//
// The next two bytes are the data length:
//
//	 l := uint16(data[1])<<8 | uint16(data[2])
//
// Bytes [3:3+l] are the string data.
//
// If tag data follows then bytes 3+l and 3+l+1 are the tag length,
// with the data following.
//
// If the import path follows, then ptrSize bytes at the end of
// the data form a *string. The import path is only set for concrete
// methods that are defined in a different package than their type.

Shrinks binary sizes:

	cmd/go: 164KB (1.6%)
	jujud:  1.0MB (1.5%)

For #6853.

Change-Id: I46b6591015b17936a443c9efb5009de8dfe8b609
Reviewed-on: https://go-review.googlesource.com/20968
Run-TryBot: David Crawshaw <crawshaw@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

The c-archive test were recently converted from shell script to Go.
Unfortunately, it also lost the ability to target iOS and Android
that lack C compilers and require exec wrappers.

Compile the c-archive test for the host and run it with the target
GOOS/GOARCH environment. Change the test to rely on go env GOOS
and go env GOARCH instead of runtime.GOOS and runtime.GOARCH.

Fixes #8345

Change-Id: I290ace2f7e96b87c55d99492feb7d660140dcb32
Reviewed-on: https://go-review.googlesource.com/21102Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

Updates #14047

Change-Id: I7e314e2c7e3e8da18ab023729740fbc9ea3f661e
Reviewed-on: https://go-review.googlesource.com/21063
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>

In f the extra & 63 is redundant because SHRQ already
looks at the bottom 6 bits only. This is a trick on AMD64
to get rid of CMPQ/SBBQ/ANDQ if one knows that the shift
counter is small.

func f(x uint64, s uint) uint64 {
        return x >> (s & 63)
}

Change-Id: I4861c902168dabec9a6a14a85750246dde94fc08
Reviewed-on: https://go-review.googlesource.com/21073Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Alexandru Moșoi <alexandru@mosoi.ro>
TryBot-Result: Gobot Gobot <gobot@golang.org>

g used to produce CMPQ/SBBQ/ANDQ, but f didn't even though
s&15 is at most s&63.

func f(x uint64, s uint) uint64 {
        return x >> (s & 63)
}
func g(x uint64, s uint) uint64 {
        return x >> (s & 15)
}

Change-Id: Iab4a1a6e10b471dead9f1203e9d894677cf07bb2
Reviewed-on: https://go-review.googlesource.com/21048
Run-TryBot: Alexandru Moșoi <alexandru@mosoi.ro>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>

The current runtime attempts to forward signals generated by non-Go
code to the original signal handler.  If it can't call the original
handler directly, it currently attempts to re-raise the signal after
resetting the handler.  In this case, the original context is lost.

This fix prevents that problem by simply returning from the go signal
handler after resetting the original handler.  It only does this when
the original handler is the system default handler, which in all cases
is known to not recover.  The signal is not reset, so it is retriggered
and the original handler takes over with the proper context.

Fixes #14899

Change-Id: Ib1c19dfa4b50d9732d7a453de3784c8141e1cbb3
Reviewed-on: https://go-review.googlesource.com/21006Reviewed-by: Ian Lance Taylor <iant@golang.org>

Android doesn't (generally) have /bin/sh.

Change-Id: I343817c342e3473d09c85155761682b5ddb043e4
Reviewed-on: https://go-review.googlesource.com/21075Reviewed-by: Ian Lance Taylor <iant@golang.org>

Fixes #14938.

Additionally some simplifications along the way.

Change-Id: I2c5fb7e32dcc6fab68fff36a49cb72e715756abe
Reviewed-on: https://go-review.googlesource.com/21046Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>

The iOS exec wrapper use complicated machinery to run a iOS binary
on a device.
Running several binaries concurrently doesn't work (reliably), which
can break tests running concurrently. For my setup, the
runtime:cpu124 and sync_cpu tests can't run reliably without one of them
crashing.

Add a file lock to the exec wrapper to serialize execution.

Fixes #14318 (for me)

Change-Id: I023610e014b327f8d66f1d2fd2e54dd0e56f2be0
Reviewed-on: https://go-review.googlesource.com/21074Reviewed-by: David Crawshaw <crawshaw@golang.org>

We were allocating a Nodes for common method calls
that did not modify the Nodes.
Though there is no clear wall time impact,
this significantly reduces the number of allocations,
so it seems worth doing.

Passes toolstash -cmp.

name       old alloc/op    new alloc/op    delta
Template      59.0MB ± 0%     58.6MB ± 0%   -0.81%        (p=0.000 n=25+25)
Unicode       41.4MB ± 0%     41.3MB ± 0%   -0.18%        (p=0.000 n=25+25)
GoTypes        198MB ± 0%      197MB ± 0%   -0.80%        (p=0.000 n=24+25)
Compiler       875MB ± 0%      865MB ± 0%   -1.09%        (p=0.000 n=25+25)

name       old allocs/op   new allocs/op   delta
Template        581k ± 0%       520k ± 0%  -10.42%        (p=0.000 n=25+25)
Unicode         413k ± 0%       403k ± 0%   -2.30%        (p=0.000 n=25+25)
GoTypes        1.78M ± 0%      1.58M ± 0%  -11.18%        (p=0.000 n=25+25)
Compiler       7.66M ± 0%      6.47M ± 0%  -15.51%        (p=0.000 n=25+25)

Change-Id: I012a9f4b333821bdf61b4f2bdff4ce5c3b5d3057
Reviewed-on: https://go-review.googlesource.com/21056Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>

CL 20892 converted the misc/cgo/testcarchive test to Go.
Unfortunately, dist does not (yet) support tests running off the host
so the testcarchive is disabled for now.

For #14318

Change-Id: Iab3d0a7b5309187a603b48f22a7fa736f089f89d
Reviewed-on: https://go-review.googlesource.com/21070Reviewed-by: David Crawshaw <crawshaw@golang.org>

A retry mechanism is in place to combat the inherent flakiness of
launching iOS test binaries. Before it covered just the starting of
lldb; expand it to cover the setup steps as well. Note that the
running of the binary itself is (still) not retried, to avoid
covering over genuine bugs.

On my test device (iPhone 5S, iOS 9.3) starting lldb can take longer
than 10 seconds, so increase the timeout for that.
Furthermore, some basic steps such as setting breakpoints in lldb
can take longer than the 1 second timeout. Increase that timeout
as well, to 2 seconds.

Finally, improve the error message for when ios-deploy is not
installed.

For #14318

Change-Id: Iba41d1bd9d023575b9454cb577b08f8cae081c2a
Reviewed-on: https://go-review.googlesource.com/21072Reviewed-by: David Crawshaw <crawshaw@golang.org>

Instruct lldb to pass through SIGCONT unhindered when running iOS
tests. Fixes the TestSIGCONT test in os/signal.

For #14318

Change-Id: I669264208cc3d6ecae9fbc8790e0b753a93a5e04
Reviewed-on: https://go-review.googlesource.com/21071Reviewed-by: David Crawshaw <crawshaw@golang.org>

For darwin/arm{,64} a non-Go thread is created to convert
EXC_BAD_ACCESS to panics. However, the Go signal handler refuse to
handle signals that would otherwise be ignored if they arrive at
non-Go threads.

Block all (posix) signals to that thread, making sure that
no unexpected signals arrive to it. At least one test, TestStop in
os/signal, depends on signals not arriving on any non-Go threads.

For #14318

Change-Id: I901467fb53bdadb0d03b0f1a537116c7f4754423
Reviewed-on: https://go-review.googlesource.com/21047Reviewed-by: David Crawshaw <crawshaw@golang.org>

Some minor scoping cleanups found by a very old version of grind.

Change-Id: I1d373817586445fc87e38305929097b652696fdd
Reviewed-on: https://go-review.googlesource.com/21064
Run-TryBot: Dave Cheney <dave@cheney.net>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Ignore superfluous trailing IDAT chunks which were not consumed when decoding
the image. This change fixes decoding of valid images in which a zero-length
IDAT chunk appears after the actual image data. It also prevents decoding of
trailing garbage IDAT chunks or maliciously embedded additional images.

Fixes #14936

Change-Id: I8c76cfa9a03496d9576f72bed2db109271f97c5e
Reviewed-on: https://go-review.googlesource.com/21045Reviewed-by: Nigel Tao <nigeltao@golang.org>

If name is /dev/{stdin,stdout,stderr}, return fileInfo.

Fixes #14853.

Change-Id: Ibf7d1ae7b9f3dc43f6ed7c905ea2c5102e1971cc
Reviewed-on: https://go-review.googlesource.com/20845Reviewed-by: Alex Brainman <alex.brainman@gmail.com>

Fixes #13996

Change-Id: I9b2c7fba0705900aca9a70bc6a2687667a9a976c
Reviewed-on: https://go-review.googlesource.com/20128Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

Remove almost all the remaining Nod(OXXX, ... ) uses. The performance
change is due entirely to the changes to func temp(*Type). The other
cleanups have no effect, as expected.

I'll address the remaining Nod(OXXX, ...) uses in a followup CL as they
are very sensitive to change.

lucky(~/go/src/cmd/compile) % benchstat /tmp/{old,new}.txt
name      old time/op    new time/op    delta
Template     391ms ± 6%     385ms ± 6%    ~     (p=0.127 n=19+20)
GoTypes      1.27s ± 2%     1.27s ± 2%    ~     (p=0.172 n=19+19)
Compiler     6.17s ± 2%     6.15s ± 2%    ~     (p=0.647 n=19+20)

name      old alloc/op   new alloc/op   delta
Template    63.7MB ± 0%    63.4MB ± 0%  -0.35%  (p=0.000 n=16+20)
GoTypes      219MB ± 0%     218MB ± 0%  -0.38%  (p=0.000 n=20+20)
Compiler     980MB ± 0%     976MB ± 0%  -0.38%  (p=0.000 n=20+20)

name      old allocs/op  new allocs/op  delta
Template      586k ± 0%      584k ± 0%  -0.30%  (p=0.000 n=20+20)
GoTypes      1.80M ± 0%     1.79M ± 0%  -0.31%  (p=0.000 n=20+20)
Compiler     7.74M ± 0%     7.71M ± 0%  -0.34%  (p=0.000 n=20+20)

Change-Id: Ie21a5443c33a23ce30f987bdddec9fe350365d35
Reviewed-on: https://go-review.googlesource.com/21017Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

This is follow-up 3 of 3 to CL 20959.

Passes toolstash -cmp.

Change-Id: I06efded21bbc970cbefa10e8f2cac1ebc6942e1b
Reviewed-on: https://go-review.googlesource.com/21054
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

This is follow-up 1 of 3 to CL 20959.

Passes toolstash -cmp.

Change-Id: I9bddf7d88333fa4755e03ff8a034a35bd01b7855
Reviewed-on: https://go-review.googlesource.com/21052
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

Fixes #14199

Change-Id: Ic9284023b663de3db1ca7b7b1e96eeab82ec0944
Reviewed-on: https://go-review.googlesource.com/21016
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>

This commit replaces some of

for i := len(x) - 1; i >= 0; i-- {...}

style loops, which do not rely on reverse iteration order.

Change-Id: I5542834286562da058200c06e7a173b13760e54d
Reviewed-on: https://go-review.googlesource.com/21044Reviewed-by: Keith Randall <khr@golang.org>

It's pretty hard to get reliable CPU numbers, even with 50 runs on an
otherwise-idle physical Linux machine, but the garbage reduction
numbers are nice. To get useful time/op numbers, I modified
compilebench to report user CPU time instead of wall time:

name       old time/op     new time/op     delta
Template       547ms ± 6%      557ms ± 5%   +1.80%        (p=0.001 n=49+49)
Unicode        360ms ± 9%      365ms ± 6%     ~           (p=0.094 n=50+45)
GoTypes        1.84s ± 3%      1.82s ± 3%   -1.50%        (p=0.000 n=50+49)
Compiler       9.19s ± 2%      9.02s ± 2%   -1.87%        (p=0.000 n=45+50)

name       old alloc/op    new alloc/op    delta
Template      63.3MB ± 0%     59.1MB ± 0%   -6.72%        (p=0.000 n=50+50)
Unicode       43.1MB ± 0%     42.9MB ± 0%   -0.47%        (p=0.000 n=50+49)
GoTypes        220MB ± 0%      200MB ± 0%   -9.00%        (p=0.000 n=50+50)
Compiler      1.00GB ± 0%     0.89GB ± 0%  -10.09%        (p=0.000 n=50+49)

name       old allocs/op   new allocs/op   delta
Template        681k ± 0%       680k ± 0%   -0.16%        (p=0.000 n=50+48)
Unicode         541k ± 0%       541k ± 0%   -0.02%        (p=0.011 n=48+50)
GoTypes        2.08M ± 0%      2.08M ± 0%   -0.19%        (p=0.000 n=48+50)
Compiler       9.24M ± 0%      9.23M ± 0%   -0.11%        (p=0.000 n=50+50)

Change-Id: I1fac4ebf85a1783e3289c3ffb1ed365442837643
Reviewed-on: https://go-review.googlesource.com/20995
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dave Cheney <dave@cheney.net>
Reviewed-by: Keith Randall <khr@golang.org>

Get rid of (*Mpint).Add's "quiet" parameter: it's always set to 0.

Inline (*Mpint).shift into (*Mpint).Lsh and (*Mpint).Rsh. There's no
need for a common shift method that can handle both left or right
shifts based on sign when the higher level abstractions only ever do
one or the other.

Change-Id: Icd3b082413f9193961b6835279e0bd4b6a6a6621
Reviewed-on: https://go-review.googlesource.com/21050
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>

Start working on arm port.  Gets close to correct
code for fibonacci:
    func fib(n int) int {
        if n < 2 {
            return n
        }
        return fib(n-1) + fib(n-2)
    }

Still a lot to do, but this is a good starting point.

Cleaned up some arch-specific dependencies in regalloc.

Change-Id: I4301c6c31a8402168e50dcfee8bcf7aee73ea9d5
Reviewed-on: https://go-review.googlesource.com/21000Reviewed-by: David Chase <drchase@google.com>

Remove reflect type information for unexported methods that do not
satisfy any interface in the program.

Ideally the unexported method would not appear in the method list at
all, but that is tricky because the slice is built by the compiler.

Reduces binary size:

	cmd/go: 81KB (0.8%)
	jujud: 258KB (0.4%)

For #6853.

Change-Id: I25ef8df6907e9ac03b18689d584ea46e7d773043
Reviewed-on: https://go-review.googlesource.com/21033Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: David Crawshaw <crawshaw@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

khr: Lifting the nil check out of the loop altogether is an admirable
goal, and this rewrite is one step on the way. But without lifting it
out of the loop, the rewrite is just hurting us.

Fixes #14917

Change-Id: Idb917f37d89f50f8e046d5ebd7c092b1e0eb0633
Reviewed-on: https://go-review.googlesource.com/21040Reviewed-by: Keith Randall <khr@golang.org>
Run-TryBot: Alexandru Moșoi <alexandru@mosoi.ro>
TryBot-Result: Gobot Gobot <gobot@golang.org>

The existing implementation for Equal and similar
functions in the bytes package operate on one byte at
at time.  This performs poorly on ppc64/ppc64le especially
when the byte buffers are large.  This change improves
those functions by loading and comparing double words where
possible.  The common code has been moved to a function
that can be shared by the other functions in this
file which perform the same type of comparison.
Further optimizations are done for the case where
>= 32 bytes are being compared.  The new function
memeqbody is used by memeq_varlen, Equal, and eqstring.

When running the bytes test with -test.bench=Equal

benchmark                     old MB/s     new MB/s     speedup
BenchmarkEqual1               164.83       129.49       0.79x
BenchmarkEqual6               563.51       445.47       0.79x
BenchmarkEqual9               656.15       1099.00      1.67x
BenchmarkEqual15              591.93       1024.30      1.73x
BenchmarkEqual16              613.25       1914.12      3.12x
BenchmarkEqual20              682.37       1687.04      2.47x
BenchmarkEqual32              807.96       3843.29      4.76x
BenchmarkEqual4K              1076.25      23280.51     21.63x
BenchmarkEqual4M              1079.30      13120.14     12.16x
BenchmarkEqual64M             1073.28      10876.92     10.13x

It was determined that the degradation in the smaller byte tests
were due to unfavorable code alignment of the single byte loop.

Fixes #14368

Change-Id: I0dd87382c28887c70f4fbe80877a8ba03c31d7cd
Reviewed-on: https://go-review.googlesource.com/20249Reviewed-by: Minux Ma <minux@golang.org>

Removes unnecessary fields from Pcln.

Change-Id: I175049ca749b510eedaf65162355bc4d7a93315e
Reviewed-on: https://go-review.googlesource.com/21041Reviewed-by: David Crawshaw <crawshaw@golang.org>

This changes how matching is done in deflate algorithm.

The major change is that we do not look for matches that are only
3 bytes in length, matches must be 4 bytes at least.
Contrary to what you would expect this actually improves the
compresion ratio, since 3 literal bytes will often be shorter
than a match after huffman encoding.
This varies a bit by source, but is most often the case when the
source is "easy" to compress.

Second of all, a "stronger" hash is used. The hash is similar to
the hashing function used by Snappy.

Overall, the speed impact is biggest on higher compression levels.
I intend to replace the "speed" compression level, which can be
seen in CL 21021.

The built-in benchmark using "digits" is slower at level 1.
I see this as an exception, since "digits" is a special type
of data, where you have low entropy (numbers 0->9), but no
significant matches. Again, CL 20021 fixes that case.

NewWriterDict is also made considerably faster, by not running data
through the entire encoder. This is not reflected by the benchmark.

Overall, the speed impact is biggest on higher compression levels.
I intend to replace the "speed" compression level.

COMPARED to tip/master:
name                       old time/op    new time/op     delta
EncodeDigitsSpeed1e4-4        401µs ± 1%      345µs ± 2%   -13.95%
EncodeDigitsSpeed1e5-4       3.19ms ± 1%     4.27ms ± 3%   +33.96%
EncodeDigitsSpeed1e6-4       27.7ms ± 4%     43.8ms ± 3%   +58.00%
EncodeDigitsDefault1e4-4      641µs ± 0%      403µs ± 1%   -37.15%
EncodeDigitsDefault1e5-4     13.8ms ± 1%      6.4ms ± 3%   -53.73%
EncodeDigitsDefault1e6-4      162ms ± 1%       64ms ± 2%   -60.51%
EncodeDigitsCompress1e4-4     627µs ± 1%      405µs ± 2%   -35.45%
EncodeDigitsCompress1e5-4    13.9ms ± 0%      6.3ms ± 2%   -54.46%
EncodeDigitsCompress1e6-4     159ms ± 1%       64ms ± 0%   -59.91%
EncodeTwainSpeed1e4-4         433µs ± 4%      331µs ± 1%   -23.53%
EncodeTwainSpeed1e5-4        2.82ms ± 1%     3.08ms ± 0%    +9.10%
EncodeTwainSpeed1e6-4        28.1ms ± 2%     28.8ms ± 0%    +2.82%
EncodeTwainDefault1e4-4       695µs ± 4%      474µs ± 1%   -31.78%
EncodeTwainDefault1e5-4      11.8ms ± 0%      7.4ms ± 0%   -37.31%
EncodeTwainDefault1e6-4       128ms ± 0%       75ms ± 0%   -40.93%
EncodeTwainCompress1e4-4      719µs ± 3%      480µs ± 0%   -33.27%
EncodeTwainCompress1e5-4     15.0ms ± 3%      8.2ms ± 2%   -45.55%
EncodeTwainCompress1e6-4      170ms ± 0%       85ms ± 1%   -49.99%

name                       old speed      new speed       delta
EncodeDigitsSpeed1e4-4     25.0MB/s ± 1%   29.0MB/s ± 2%   +16.24%
EncodeDigitsSpeed1e5-4     31.4MB/s ± 1%   23.4MB/s ± 3%   -25.34%
EncodeDigitsSpeed1e6-4     36.1MB/s ± 4%   22.8MB/s ± 3%   -36.74%
EncodeDigitsDefault1e4-4   15.6MB/s ± 0%   24.8MB/s ± 1%   +59.11%
EncodeDigitsDefault1e5-4   7.27MB/s ± 1%  15.72MB/s ± 3%  +116.23%
EncodeDigitsDefault1e6-4   6.16MB/s ± 0%  15.60MB/s ± 2%  +153.25%
EncodeDigitsCompress1e4-4  15.9MB/s ± 1%   24.7MB/s ± 2%   +54.97%
EncodeDigitsCompress1e5-4  7.19MB/s ± 0%  15.78MB/s ± 2%  +119.62%
EncodeDigitsCompress1e6-4  6.27MB/s ± 1%  15.65MB/s ± 0%  +149.52%
EncodeTwainSpeed1e4-4      23.1MB/s ± 4%   30.2MB/s ± 1%   +30.68%
EncodeTwainSpeed1e5-4      35.4MB/s ± 1%   32.5MB/s ± 0%    -8.34%
EncodeTwainSpeed1e6-4      35.6MB/s ± 2%   34.7MB/s ± 0%    -2.77%
EncodeTwainDefault1e4-4    14.4MB/s ± 4%   21.1MB/s ± 1%   +46.48%
EncodeTwainDefault1e5-4    8.49MB/s ± 0%  13.55MB/s ± 0%   +59.50%
EncodeTwainDefault1e6-4    7.83MB/s ± 0%  13.25MB/s ± 0%   +69.19%
EncodeTwainCompress1e4-4   13.9MB/s ± 3%   20.8MB/s ± 0%   +49.83%
EncodeTwainCompress1e5-4   6.65MB/s ± 3%  12.20MB/s ± 2%   +83.51%
EncodeTwainCompress1e6-4   5.88MB/s ± 0%  11.76MB/s ± 1%  +100.06%

Change-Id: I724e33c1dd3e3a6a1b0a68e094baa959352baf32
Reviewed-on: https://go-review.googlesource.com/20929
Run-TryBot: Nigel Tao <nigeltao@golang.org>
Reviewed-by: Nigel Tao <nigeltao@golang.org>