cmd/go: default to GOPROXY=https://proxy.golang.org and GOSUMDB=sum.golang.org

This CL changes the default module download and module verification mechanisms
to use the Go module mirror and Go checksum database run by Google.
See https://proxy.golang.org/privacy for the services' privacy policy.
(Today, that URL is a redirect to Google's standard privacy policy,
which covers these services as well. If we publish a more specific
privacy policy just for these services, that URL will be updated to
display or redirect to it.)

See 'go help modules' and 'go help modules-auth' for details (added in this CL).

To disable the mirror and checksum database for non-public modules:

	go env -w GONOPROXY=*.private.net,your.com/*
	go env -w GONOSUMDB=*.private.net,your.com/*

(If you are using a private module proxy then you'd only do the second.)

If you run into problems with the behavior of the go command when using
the Go module mirror or the Go checksum database, please file issues at
https://golang.org/issue/new, so that we can address them for the
Go 1.13 release.

For #25530.

This CL also documents GONOPROXY.
Fixes #32056.

Change-Id: I2fde82e071742272b0842efd9580df1a56947fec
Reviewed-on: https://go-review.googlesource.com/c/go/+/178179
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>

src/cmd/go/alldocs.go

@@ -2020,21 +2020,6 @@
 //
 // Module proxy protocol
 //
-// The go command by default downloads modules from version control systems
-// directly, just as 'go get' always has. The GOPROXY environment variable allows
-// further control over the download source. If GOPROXY is unset, is the empty string,
-// or is the string "direct", downloads use the default direct connection to version
-// control systems. Setting GOPROXY to "off" disallows downloading modules from
-// any source. Otherwise, GOPROXY is expected to be a comma-separated list of
-// the URLs of module proxies, in which case the go command will fetch modules
-// from those proxies. For each request, the go command tries each proxy in sequence,
-// only moving to the next if the current proxy returns a 404 or 410 HTTP response.
-// The string "direct" may appear in the proxy list, to cause a direct connection to
-// be attempted at that point in the search.
-//
-// No matter the source of the modules, downloaded modules must match existing
-// entries in go.sum (see 'go help modules' for discussion of verification).
-//
 // A Go module proxy is any web server that can respond to GET requests for
 // URLs of a specified form. The requests have no query parameters, so even
 // a site serving from a fixed file system (including a file:/// URL)
@@ -2591,16 +2576,43 @@
 //
 // Module downloading and verification
 //
-// The go command checks downloads against known checksums,
-// to detect unexpected changes in the content of any specific module
-// version from one day to the next. See 'go help module-auth' for details.
+// The go command can fetch modules from a proxy or connect to source control
+// servers directly, according to the setting of the GOPROXY environment
+// variable (see 'go help env'). The default setting for GOPROXY is
+// "https://proxy.golang.org", the Go module mirror run by Google.
+// See https://proxy.golang.org/privacy for the service's privacy policy.
+// If GOPROXY is set to the string "direct", downloads use a direct connection
+// to source control servers. Setting GOPROXY to "off" disallows downloading
+// modules from any source. Otherwise, GOPROXY is expected to be a comma-separated
+// list of the URLs of module proxies, in which case the go command will fetch
+// modules from those proxies. For each request, the go command tries each proxy
+// in sequence, only moving to the next if the current proxy returns a 404 or 410
+// HTTP response. The string "direct" may appear in the proxy list,
+// to cause a direct connection to be attempted at that point in the search.
+// Any proxies listed after "direct" are never consulted.
+//
+// The GONOPROXY environment variable is a comma-separated list of
+// glob patterns (in the syntax of Go's path.Match) of module path prefixes
+// that should always be fetched directly, ignoring the GOPROXY setting.
+// For example,
+//
+// 	GONOPROXY=*.corp.example.com,rsc.io/private
+//
+// forces a direct connection to download modules with path prefixes matching
+// either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private",
+// and "rsc.io/private/quux".
 //
-// The go command can fetch modules from a proxy instead of connecting
-// to source control systems directly, according to the setting of the GOPROXY
-// environment variable.
+// The 'go env -w' command (see 'go help env') can be used to set these variables
+// for future go command invocations.
 //
-// See 'go help goproxy' for details about the proxy and also the format of
-// the cached downloaded packages.
+// No matter the source of the modules, the go command checks downloads against
+// known checksums, to detect unexpected changes in the content of any specific
+// module version from one day to the next. This check first consults the current
+// module's go.sum file but falls back to the Go checksum database.
+// See 'go help module-auth' for details.
+//
+// See 'go help goproxy' for details about the proxy protocol and also
+// the format of the cached downloaded packages.
 //
 // Modules and vendoring
 //
@@ -2778,18 +2790,17 @@
 // database requires giving the public key explicitly. The URL defaults to
 // "https://" followed by the database name.
 //
-// GOSUMDB defaults to "sum.golang.org" when GOPROXY="https://proxy.golang.org"
-// and otherwise defaults to "off". NOTE: The GOSUMDB will later default to
-// "sum.golang.org" unconditionally.
+// GOSUMDB defaults to "sum.golang.org", the Go checksum database run by Google.
+// See https://sum.golang.org/privacy for the service's privacy policy.
 //
 // If GOSUMDB is set to "off", or if "go get" is invoked with the -insecure flag,
-// the checksum database is never consulted, but at the cost of giving up the
-// security guarantee of verified repeatable downloads for all modules.
-// A better way to bypass the checksum database for specific modules is
-// to use the GONOSUMDB environment variable.
+// the checksum database is not consulted, and all unrecognized modules are
+// accepted, at the cost of giving up the security guarantee of verified repeatable
+// downloads for all modules. A better way to bypass the checksum database
+// for specific modules is to use the GONOSUMDB environment variable.
 //
 // The GONOSUMDB environment variable is a comma-separated list of
-// patterns (in the syntax of Go's path.Match) of module path prefixes
+// glob patterns (in the syntax of Go's path.Match) of module path prefixes
 // that should not be compared against the checksum database.
 // For example,
 //
@@ -2799,6 +2810,9 @@
 // either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private",
 // and "rsc.io/private/quux".
 //
+// The 'go env -w' command (see 'go help env') can be used to set these variables
+// for future go command invocations.
+//
 //
 // Testing flags
 //

src/cmd/go/internal/cfg/cfg.go

@@ -303,13 +303,6 @@ func goproxy() string {
 		return v
 	}
 
-	// Proxy is off by default for now.
-	// TODO(rsc): Remove this condition, turning it on always.
-	// (But do NOT do this without approval from rsc.)
-	if true {
-		return "direct"
-	}
-
 	return "https://proxy.golang.org"
 }
 
@@ -319,13 +312,6 @@ func gosumdb() string {
 		return v
 	}
 
-	// Checksum database is off by default except when GOPROXY is proxy.golang.org.
-	// TODO(rsc): Remove this condition, turning it on always.
-	// (But do NOT do this without approval from rsc.)
-	if !strings.HasPrefix(GOPROXY, "https://proxy.golang.org") {
-		return "off"
-	}
-
 	return "sum.golang.org"
 }
 

src/cmd/go/internal/modfetch/fetch.go

@@ -702,18 +702,17 @@ The go command knows the public key of sum.golang.org; use of any other
 database requires giving the public key explicitly. The URL defaults to
 "https://" followed by the database name.
 
-GOSUMDB defaults to "sum.golang.org" when GOPROXY="https://proxy.golang.org"
-and otherwise defaults to "off". NOTE: The GOSUMDB will later default to
-"sum.golang.org" unconditionally.
+GOSUMDB defaults to "sum.golang.org", the Go checksum database run by Google.
+See https://sum.golang.org/privacy for the service's privacy policy.
 
 If GOSUMDB is set to "off", or if "go get" is invoked with the -insecure flag,
-the checksum database is never consulted, but at the cost of giving up the
-security guarantee of verified repeatable downloads for all modules.
-A better way to bypass the checksum database for specific modules is
-to use the GONOSUMDB environment variable.
+the checksum database is not consulted, and all unrecognized modules are
+accepted, at the cost of giving up the security guarantee of verified repeatable
+downloads for all modules. A better way to bypass the checksum database
+for specific modules is to use the GONOSUMDB environment variable.
 
 The GONOSUMDB environment variable is a comma-separated list of
-patterns (in the syntax of Go's path.Match) of module path prefixes
+glob patterns (in the syntax of Go's path.Match) of module path prefixes
 that should not be compared against the checksum database.
 For example,
 
@@ -722,5 +721,8 @@ For example,
 disables checksum database lookups for modules with path prefixes matching
 either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private",
 and "rsc.io/private/quux".
+
+The 'go env -w' command (see 'go help env') can be used to set these variables
+for future go command invocations.
 `,
 }

src/cmd/go/internal/modfetch/proxy.go

@@ -32,21 +32,6 @@ var HelpGoproxy = &base.Command{
 	UsageLine: "goproxy",
 	Short:     "module proxy protocol",
 	Long: `
-The go command by default downloads modules from version control systems
-directly, just as 'go get' always has. The GOPROXY environment variable allows
-further control over the download source. If GOPROXY is unset, is the empty string,
-or is the string "direct", downloads use the default direct connection to version
-control systems. Setting GOPROXY to "off" disallows downloading modules from
-any source. Otherwise, GOPROXY is expected to be a comma-separated list of
-the URLs of module proxies, in which case the go command will fetch modules
-from those proxies. For each request, the go command tries each proxy in sequence,
-only moving to the next if the current proxy returns a 404 or 410 HTTP response.
-The string "direct" may appear in the proxy list, to cause a direct connection to
-be attempted at that point in the search.
-
-No matter the source of the modules, downloaded modules must match existing
-entries in go.sum (see 'go help modules' for discussion of verification).
-
 A Go module proxy is any web server that can respond to GET requests for
 URLs of a specified form. The requests have no query parameters, so even
 a site serving from a fixed file system (including a file:/// URL)

src/cmd/go/internal/modload/help.go

@@ -328,16 +328,43 @@ module file trees.
 
 Module downloading and verification
 
-The go command checks downloads against known checksums,
-to detect unexpected changes in the content of any specific module
-version from one day to the next. See 'go help module-auth' for details.
-
-The go command can fetch modules from a proxy instead of connecting
-to source control systems directly, according to the setting of the GOPROXY
-environment variable.
-
-See 'go help goproxy' for details about the proxy and also the format of
-the cached downloaded packages.
+The go command can fetch modules from a proxy or connect to source control
+servers directly, according to the setting of the GOPROXY environment
+variable (see 'go help env'). The default setting for GOPROXY is
+"https://proxy.golang.org", the Go module mirror run by Google.
+See https://proxy.golang.org/privacy for the service's privacy policy.
+If GOPROXY is set to the string "direct", downloads use a direct connection
+to source control servers. Setting GOPROXY to "off" disallows downloading
+modules from any source. Otherwise, GOPROXY is expected to be a comma-separated
+list of the URLs of module proxies, in which case the go command will fetch
+modules from those proxies. For each request, the go command tries each proxy
+in sequence, only moving to the next if the current proxy returns a 404 or 410
+HTTP response. The string "direct" may appear in the proxy list,
+to cause a direct connection to be attempted at that point in the search.
+Any proxies listed after "direct" are never consulted.
+
+The GONOPROXY environment variable is a comma-separated list of
+glob patterns (in the syntax of Go's path.Match) of module path prefixes
+that should always be fetched directly, ignoring the GOPROXY setting.
+For example,
+
+	GONOPROXY=*.corp.example.com,rsc.io/private
+
+forces a direct connection to download modules with path prefixes matching
+either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private",
+and "rsc.io/private/quux".
+
+The 'go env -w' command (see 'go help env') can be used to set these variables
+for future go command invocations.
+
+No matter the source of the modules, the go command checks downloads against
+known checksums, to detect unexpected changes in the content of any specific
+module version from one day to the next. This check first consults the current
+module's go.sum file but falls back to the Go checksum database.
+See 'go help module-auth' for details.
+
+See 'go help goproxy' for details about the proxy protocol and also
+the format of the cached downloaded packages.
 
 Modules and vendoring
 

src/cmd/go/testdata/script/mod_sumdb_golang.txt

-[!net] skip
-
+# Test default GOPROXY and GOSUMDB
 env GOPROXY=
 env GOSUMDB=
 go env GOPROXY
-stdout '^direct$'
+stdout '^https://proxy.golang.org$'
 go env GOSUMDB
-stdout '^off$'
+stdout '^sum.golang.org$'
 env GOPROXY=https://proxy.golang.org
 go env GOSUMDB
 stdout '^sum.golang.org$'
 
 # download direct from github
+[!net] skip
 env GOSUMDB=sum.golang.org
 env GOPROXY=direct
 go get -m rsc.io/quote