Avoids false positive pointer arithmetic panic.

Fixes #35027.

Change-Id: Idd008caaab25fcf739327ac50a021b835ef13def
Reviewed-on: https://go-review.googlesource.com/c/go/+/202560
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>