net/http: add Transport.ReadBufferSize and WriteBufferSize

Previously transport was using the hardcoded bufio.defaultBufSize
(4096), limiting throughput and increasing cpu usage when uploading or
downloading large files.

Add options to allow users to configure the buffer sizes as needed.

I tested the maximum benefit of this change by uploading data from
/dev/zero to a server discarding the bytes. Here is an example upload
using the default buffer size:

$ time ./upload 10 https://localhost:8000/
Uploaded 10.00g in 25.13 seconds (407.49m/s)

real	0m25.135s
user	0m5.167s
sys	0m11.643s

With this change, using 128k buffer size:

$ time ./upload 10 https://localhost:8000/
Uploaded 10.00g in 7.93 seconds (1291.51m/s)

real	0m7.935s
user	0m4.517s
sys	0m2.603s

In real world usage the difference will be smaller, depending on the
local and remote storage and the network.

See https://github.com/nirs/http-bench for more info.

Fixes #22618

Change-Id: Iac99ed839c7b95d6dc66602ba8fe1fc5b500c47c
Reviewed-on: https://go-review.googlesource.com/c/go/+/76410Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

src/net/http/transport.go

@@ -254,6 +254,16 @@ type Transport struct {
 	// Zero means to use a default limit.
 	MaxResponseHeaderBytes int64
 
+	// WriteBufferSize specifies the size of the write buffer used
+	// when writing to the transport.
+	// If zero, a default (currently 4KB) is used.
+	WriteBufferSize int
+
+	// ReadBufferSize specifies the size of the read buffer used
+	// when reading from the transport.
+	//If zero, a default (currently 4KB) is used.
+	ReadBufferSize int
+
 	// nextProtoOnce guards initialization of TLSNextProto and
 	// h2transport (via onceSetNextProtoDefaults)
 	nextProtoOnce sync.Once
@@ -266,6 +276,20 @@ type Transport struct {
 	ForceAttemptHTTP2 bool
 }
 
+func (t *Transport) writeBufferSize() int {
+	if t.WriteBufferSize > 0 {
+		return t.WriteBufferSize
+	}
+	return 4 << 10
+}
+
+func (t *Transport) readBufferSize() int {
+	if t.ReadBufferSize > 0 {
+		return t.ReadBufferSize
+	}
+	return 4 << 10
+}
+
 // h2Transport is the interface we expect to be able to call from
 // net/http against an *http2.Transport that's either bundled into
 // h2_bundle.go or supplied by the user via x/net/http2.
@@ -1360,8 +1384,9 @@ func (t *Transport) dialConn(ctx context.Context, cm connectMethod) (*persistCon
 	if t.MaxConnsPerHost > 0 {
 		pconn.conn = &connCloseListener{Conn: pconn.conn, t: t, cmKey: pconn.cacheKey}
 	}
-	pconn.br = bufio.NewReader(pconn)
-	pconn.bw = bufio.NewWriter(persistConnWriter{pconn})
+	pconn.br = bufio.NewReaderSize(pconn, t.readBufferSize())
+	pconn.bw = bufio.NewWriterSize(persistConnWriter{pconn}, t.writeBufferSize())
+
 	go pconn.readLoop()
 	go pconn.writeLoop()
 	return pconn, nil