x/website: `Internal Server Error` from `go-get` requests to golang.org

[bcmills]

#!watchflakes
post <- log ~ `https://golang\.org/.*: 500 Internal Server Error`

            > ! go list -m golang.org/x/text
            [stderr]
            go: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 500 Internal Server Error
            go: example.com@v0.0.0 requires
            	golang.org/x/text@v0.1.1-0.20170915032832-14c0d48ead0c+incompatible: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 500 Internal Server Error
            [exit status 1]

greplogs -l -e 'https://golang\.org/x/[^?]*\?go-get=1.* 5\d\d ' --since=2021-01-01
2022-05-10T20:13:31-2ecf747/linux-amd64-longtest
2022-05-10T16:17:51-30a2750/linux-386-longtest
2022-05-10T15:33:47-f7209f9/linux-386-longtest
2022-05-04T20:20:28-fbb47e8/linux-amd64-longtest
2022-04-11T02:55:52-a6f6932/linux-386-longtest
2022-02-15T21:18:59-08ed488/linux-386-longtest
2022-01-11T15:23:23-28f2869/linux-386-longtest
2022-01-06T21:38:57-c5540e5/linux-amd64-longtest
2021-11-22T23:09:35-9678f79/linux-386-longtest
2021-11-09T16:55:47-34abc12/linux-386-longtest
2021-10-15T16:57:36-a80e53e/linux-386-longtest
2021-10-08T21:38:44-74abcab/linux-386-longtest
2021-10-04T19:51:37-579ff8b/linux-amd64-longtest
2021-09-24T00:30:36-c90ead9/linux-386-longtest
2021-09-02T13:52:06-a35c5c9/linux-386-longtest
2021-08-28T18:48:37-f4cd001/linux-386-longtest
2021-04-21T13:38:59-4d56576/linux-386-longtest

[bcmills]

@golang/release: the cluster of status-500 responses yesterday in particular seems a bit worrying. Did those show up in internal monitoring?

[gopherbot]

Change https://go.dev/cl/405714 mentions this issue: cmd/go: add timestamps to script test output

[dmitshur]

There doesn't appear to be anything abnormal for yesterday at https://console.cloud.google.com/appengine?project=golang-org nor https://console.cloud.google.com/monitoring/uptime?project=golang-org (which you should have access to). However, I am seeing an increased latency as of May 9, 8 pm at https://console.cloud.google.com/monitoring/uptime/go-dev?project=golang-org.

Also CC @jamalc. (The tools team is the the go.dev website owner now.)

[bcmills]

@dmitshur, the uptime charts don't seem detailed enough to surface these errors, and I don't appear to have the right permissions for the /appengine page. (Is there a URL for debug logs somewhere? Or is the /appengine page where the debug logs are supposed to be?)

[bcmills]

(https://build.golang.org/log/932d64b5e72fd57cce74a37d3c43531068766e0a shows one of these failures with an approximate timestamp of 2022-05-11T19:47:45Z.)

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `https://golang\.org/.*: 500 Internal Server Error`

2023-02-10 17:29 linux-386-longtest go@aa5b2255 cmd/go.TestDefaultGOPATHGet (log)

vcs-test.golang.org rerouted to http://127.0.0.1:46319
https://vcs-test.golang.org rerouted to https://127.0.0.1:39907
go test proxy running at GOPROXY=http://127.0.0.1:37931/mod
2023/02/10 18:13:29 http: TLS handshake error from 127.0.0.1:39924: EOF
2023/02/10 18:13:29 http: TLS handshake error from 127.0.0.1:39878: EOF
2023/02/10 18:13:30 http: TLS handshake error from 127.0.0.1:39894: EOF
2023/02/10 18:13:30 http: TLS handshake error from 127.0.0.1:39968: EOF
2023/02/10 18:13:30 http: TLS handshake error from 127.0.0.1:39928: EOF
2023/02/10 18:13:31 http: TLS handshake error from 127.0.0.1:40030: read tcp 127.0.0.1:39907->127.0.0.1:40030: read: connection reset by peer
2023/02/10 18:13:31 http: TLS handshake error from 127.0.0.1:40010: read tcp 127.0.0.1:39907->127.0.0.1:40010: read: connection reset by peer
2023/02/10 18:13:31 http: TLS handshake error from 127.0.0.1:40036: EOF
2023/02/10 18:13:31 http: TLS handshake error from 127.0.0.1:40044: EOF
2023/02/10 18:13:31 http: TLS handshake error from 127.0.0.1:40102: EOF
2023/02/10 18:13:31 http: TLS handshake error from 127.0.0.1:40118: EOF
2023/02/10 18:13:31 http: TLS handshake error from 127.0.0.1:40076: EOF
2023/02/10 18:13:31 http: TLS handshake error from 127.0.0.1:40120: EOF
2023/02/10 18:13:31 http: TLS handshake error from 127.0.0.1:40086: EOF
--- FAIL: TestDefaultGOPATHGet (1.47s)
    go_test.go:1396: running testgo [get -v github.com/golang/example/hello]
    go_test.go:1396: standard error:
    go_test.go:1396: github.com/golang/example (download)
        created GOPATH=/workdir/tmp/cmd-go-test-475988687/gotest2175577613/home/go; see 'go help gopath'
        package golang.org/x/example/stringutil: unrecognized import path "golang.org/x/example/stringutil": reading https://golang.org/x/example/stringutil?go-get=1: 500 Internal Server Error

    go_test.go:1396: go [get -v github.com/golang/example/hello] failed unexpectedly in /workdir/go/src/cmd/go: exit status 1

2023-02-10 17:29 linux-amd64-longtest go@aa5b2255 cmd/go/internal/modload.TestQueryImport (log)

vcs-test.golang.org rerouted to http://127.0.0.1:33193
https://vcs-test.golang.org rerouted to https://127.0.0.1:33741
go: finding module for package golang.org/x/net/context
go: finding module for package golang.org/x/net
go: finding module for package golang.org/x/text
go: finding module for package github.com/rsc/quote/buggy
go: finding module for package github.com/rsc/quote
go: finding module for package golang.org/x/foo/bar
--- FAIL: TestQueryImport (15.24s)
    --- FAIL: TestQueryImport/golang.org_x_net_context (13.45s)
        import_test.go:81: queryImport(_, "golang.org/x/net/context"): cannot find module providing package golang.org/x/net/context: unrecognized import path "golang.org/x/net/context": reading https://golang.org/x/net/context?go-get=1: 500 Internal Server Error
    --- FAIL: TestQueryImport/golang.org_x_net (0.00s)
        import_test.go:88: queryImport(_, "golang.org/x/net"): error "cannot find module providing package golang.org/x/net: unrecognized import path \"golang.org/x/net\": reading https://golang.org/x/net?go-get=1: 500 Internal Server Error", want error matching `module golang.org/x/net@.* found \(v[01]\.\d+\.\d+\), but does not contain package golang.org/x/net`
    --- FAIL: TestQueryImport/golang.org_x_text (0.05s)
        import_test.go:81: queryImport(_, "golang.org/x/text"): cannot find module providing package golang.org/x/text: unrecognized import path "golang.org/x/text": reading https://golang.org/x/text?go-get=1: 500 Internal Server Error

2023-02-10 17:29 linux-386-longtest go@1fa2deb1 cmd/go.TestDefaultGOPATHGet (log)

vcs-test.golang.org rerouted to http://127.0.0.1:37021
https://vcs-test.golang.org rerouted to https://127.0.0.1:34275
go test proxy running at GOPROXY=http://127.0.0.1:43083/mod
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:55930: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:55908: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:55914: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:55958: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:55948: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:55970: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:56008: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:56016: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:56066: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:56082: EOF
2023/02/10 18:13:47 http: TLS handshake error from 127.0.0.1:56084: EOF
--- FAIL: TestDefaultGOPATHGet (3.14s)
    go_test.go:1396: running testgo [get -v github.com/golang/example/hello]
    go_test.go:1396: standard error:
    go_test.go:1396: github.com/golang/example (download)
        created GOPATH=/workdir/tmp/cmd-go-test-3740005368/gotest524644450/home/go; see 'go help gopath'
        package golang.org/x/example/stringutil: unrecognized import path "golang.org/x/example/stringutil": reading https://golang.org/x/example/stringutil?go-get=1: 500 Internal Server Error

    go_test.go:1396: go [get -v github.com/golang/example/hello] failed unexpectedly in /workdir/go/src/cmd/go: exit status 1

2023-02-10 17:41 linux-amd64-longtest go@7628627c cmd/go.TestDefaultGOPATHGet (log)

vcs-test.golang.org rerouted to http://127.0.0.1:38245
https://vcs-test.golang.org rerouted to https://127.0.0.1:46611
go test proxy running at GOPROXY=http://127.0.0.1:38755/mod
2023/02/10 18:12:17 http: TLS handshake error from 127.0.0.1:41582: read tcp 127.0.0.1:46611->127.0.0.1:41582: read: connection reset by peer
2023/02/10 18:12:17 http: TLS handshake error from 127.0.0.1:41572: read tcp 127.0.0.1:46611->127.0.0.1:41572: read: connection reset by peer
2023/02/10 18:12:18 http: TLS handshake error from 127.0.0.1:41610: EOF
2023/02/10 18:12:18 http: TLS handshake error from 127.0.0.1:41604: EOF
2023/02/10 18:12:18 http: TLS handshake error from 127.0.0.1:46964: read tcp 127.0.0.1:46611->127.0.0.1:46964: read: connection reset by peer
2023/02/10 18:12:18 http: TLS handshake error from 127.0.0.1:46996: read tcp 127.0.0.1:46611->127.0.0.1:46996: read: connection reset by peer
2023/02/10 18:12:18 http: TLS handshake error from 127.0.0.1:46974: EOF
--- FAIL: TestDefaultGOPATHGet (11.62s)
    go_test.go:1396: running testgo [get -v github.com/golang/example/hello]
    go_test.go:1396: standard error:
    go_test.go:1396: github.com/golang/example (download)
        created GOPATH=/workdir/tmp/cmd-go-test-1318724607/gotest1419479580/home/go; see 'go help gopath'
        package golang.org/x/example/stringutil: unrecognized import path "golang.org/x/example/stringutil": reading https://golang.org/x/example/stringutil?go-get=1: 500 Internal Server Error

    go_test.go:1396: go [get -v github.com/golang/example/hello] failed unexpectedly in /workdir/go/src/cmd/go: exit status 1

— watchflakes