cmd/go: add package forwarding

[matloob]

Overview:

The aim of this discussion is talk about a potential proposal to add a mechanism to add forwarding declarations from one package to another package. When the go build system encounters a forwarding declaration on a package, it replaces all imports of that package in the build with imports of the forwarded package. A package that has been moved or forked to a different module can forward itself to the destination or fork, +incompatible versions of modules can forward their packages to versions with major version suffixes, and GitHub repository modules that have changed their name can forward themselves to their new name.

Proposal:

We add a forwarding directive that would be added to a package statement. As an example, consider the package example.com/from/p with the following package declaration:

//go:forward example.com/to/q
package p

The //go:forward declaration tells the Go build system that all imports of the import path example.com/from/p should be treated as imports of the import path example.com/to/q. (This proposal targets the Go build system; if it is accepted, other build systems, such as Bazel would need to implement special support for forwarding to allow importing packages using forwarding directives.) Any uses of the import example.com/from/p that are found would be internally rewritten as imports of example.com/to/q. [We internally would create a loadPkg for example.com/from/p that records that it forwards to example.com/to/q] Packages are allowed to forward to internal packages if they would be able to import those packages.

When the build system encounters a //go:forward declaration, it also enforces that the package (after build constraint resolution) does not contain any code.

The go list output for a forwarded package would contain a new Forward field for forwarded packages that points to the package it's forwarded to. Tools that make use of go list output such as those based on golang.org/x/tools/go/packages would need to be updated to understand the forwarding directives and forward to the target package.

We will also add a subcommand to the go command to aid in adding forwarding declarations:

go forward <directory path> <destination path>[/...] will either create new packages with forwarding declarations, or add forwarding declarations to already existing packages, pointing to the destination path. The /… pattern, can only appear at the end of the destination path, will recursively forward subdirectories directory path to the corresponding subdirectories of the destination package's directory. It will not cross module boundaries.
If the from directories already exist, go forward would place the forwarding declarations in a new go file forward.go and delete all source files (Go and non-Go) in the directory that are recognized by the Go build system.

Open question: we don't allow declarations in the same package, should we allow declarations in tests? probably not

Use cases:

Standard use case:

Module example.com/old stops development and development moves to example.com/new. Forwarding declarations are added to all the packages of example.com/old on the last published version of that package using

go forward example.com/old example.com/new

. When a user tries to update example.com/old to the latest version, the forwarding declarations are pulled in from that last published version and all references to the old path are rewritten to point to the new path, "upgrading" the module to the new module. A requirement to the new module is pulled in.

The user can also add requirements on all 'old' module versions forwarding to the new module in the module being forwarded to. We could do this via a test that the old import works. Adding the requirements ensures that if there are imports of the old module in any dependencies of users of the new module, the forwarding declarations are seen and forward those to the new module. That will mean the new module packages are used uniformly for both import paths.

Old module is no longer maintained, support a migration:

In this use case, module example.com/old stops development, but they haven't picked a successor module. A fork of the module example.com/new wants to be able to support a migration of the usages of the module to the new path using a mix of the old and new paths in the interim. They could release an 0.1.0 version of example.com/new that forwards all corresponding packages to the old module. Then users of the old module can add a requirement on v0.1.0 of the new module, and slowly migrate their imports to the old module to point to paths in the new module. Because of the forwarding declarations in v0.1.0, both imports of the old and new module would be uniformly resolved to the old module. This would allow imports to be migrated piecemeal until all of them become imports of the new module. Once that is the case, the requirements can be upgraded to a v1.0.0 version of the new module that no longer has forwarding declaration, and perhaps introduces new functionality/fixes/etc. The author of the module could additionally include the new code in v0.1.0, but under a build constraint, for example under the tag examplecomnew. Then users could run go test -tags=examplecomnew to verify the individual migrations before they upgrade to 1.0.0.

Github Repository renamed with the same major version.

Module github.com/olduser/oldmodule is renamed to github.com/newuser/newmodule. To facilitate this, a "final" version of the module is released that declares itself as github.com/olduser/oldmodule in the go.mod module declaration. This version will forward all packages to their corresponding package in github.com/newuser/newmodule. And then a following version is released (without the forwarding declarations, but instead the normal package code) that declares itself as github.com/newuser/newmodule in its go.mod file. An upgrade of github.com/olduser/oldmodule would upgrade the module to the latest version without errors, which would be the version with the forwarding declaration (That's because upgrade upgrades to the latest version without errors, which doesn't include any of the versions with the new module path because they are module name mismatches with the module requested. It is important to note that this behavior can be O(N) with the number of releases of the new module at the same major version because each version is checked for errors in order going down until one without errors is found). That would in turn add forwarding declarations to the new module which would introduce a requirement on the new module, so that we'd upgrade to the latest version of the new module.

Allowing replacements to point to other active modules (Issue #26904)

In this use case, module example.com/old is forked to example.com/new, but the owners of example.com/old don't want to add forwarding declarations to example.com/new. The user of example.com/new wants all uses of example.com/old in their transitive dependencies to point to example.com/new. They can't just do a replace to example.com/new because a replacement can't point to another module that's also in use in the module's transitive dependencies. But what they can do is replace example.com/old with a module that has forwarding declarations corresponding to all of example.com/old's packages. Then all uses of example.com/old will be replaced and then forwarded to example.com/new.

[willfaught]

When the go build system encounters a forwarding declaration on a package, it replaces all imports of that package in the build with imports of the forwarded package.

Is it like aliases, where the old and new are interchangeable, and can be used together in the same package?

+incompatible versions of modules can forward their packages to versions with major version suffixes

Can you remind us what +incompatible versions are?

go forward would place the forwarding declarations in a new go file forward.go

You mean forwarding declaration, singular, correct?

(or forwardN.go for an N where the filename doesn't exist)

Given

When the build system encounters a //go:forward declaration, it also enforces that the package (after build constraint resolution) does not contain any code.

why would forward.go already exist? Wouldn't it be the only file left in the old package? I'm not clear on why a subcommand is useful, if all we need is a single file with a special package comment.

Open question: we don't allow declarations in the same package, should we allow declarations in tests? probably not

I don't understand what this is saying, or asking.

[bcmills]

Can you remind us what +incompatible versions are?

https://go.dev/ref/mod#incompatible-versions

[bcmills]

You mean forwarding declaration, singular, correct?

Yes.

[bcmills]

Open question: we don't allow declarations in the same package, should we allow declarations in tests? probably not

I don't understand what this is saying, or asking.

A package containing a //go:forward directive must not also declare types, functions, etc, because that would make the two packages different. But can it declare Test functions in a _test.go file? Those would be in the test variant, not the package itself.

[willfaught]

@bcmills Thanks. There's one left, do you know the answer?

why would forward.go already exist? Wouldn't it be the only file left in the old package? I'm not clear on why a subcommand is useful, if all we need is a single file with a special package comment.

[bcmills]

The point of the go forward subcommand is to make it easy to script for large numbers of packages, such as when forwarding an entire former repo location to a new repo location.

(You could write an external script to do the same thing, I suppose, but we expect a significant fraction of Go users to need it at some point.)

[willfaught]

Can you provide real-world examples where this would have been/would be useful?

[oxisto]

We (golang-jwt) actually also had the issue that we needed to "forward" from an +incompatible version to a module-tagged version, which created major confusion at first and led problems that old incompatible versions "spilled" into the new repository, which we wanted to avoid (see #46440 (comment))

[hyangah]

Thanks @bcmills for the examples. This package-by-package migration idea under this discussion is interesting and I see it can be useful in certain cases(*). Meanwhile, however, I am afraid it looks complex for human to understand/debug.

Reading each referenced case,

I think module-by-module replacement (as originally asked in #26904) would be sufficient for both golang-jwt/jwt#37 and etcd-io/etcd#9965 (not the gradual package-to-package replacement as described in this discussion).

An interesting case is kubernetes/kubernetes#104007 which was the motivation for etcd-io/etcd#13254 where multiple forks of jwt-go were involved back then, as direct (from k8s) and transitive dependencies (from etcd, heketi, etc). If k8s maintainers wanted to utilize the current proposed idea to migrate before other dependencies also completed migrations eventually, they would've needed to create their own forks of the forks of jwt-go for replace-purpose to preserve module paths and set the forward rules in their own forks package by package. (I don't see why package-by-package transition is necessary in this case yet) Am I understanding this correctly?

(etcd-io/etcd#12068 case is too complex because it involves too many things - vanity import paths, glide.lock, pre-module, and multiple transitive modules, and old go, and intention to support old go :-()

(*) example where package-to-package migration are useful: assume we want to forward golang.org/x/exp/slog imports to the new slog package in the std module, the module boundaries don't match.

[bcmills]

If k8s maintainers wanted to utilize the current proposed idea to migrate before other dependencies also completed migrations eventually, they would've needed to create their own forks of the forks of jwt-go for replace-purpose to preserve module paths and set the forward rules in their own forks package by package.

Correct — although if there is a specific project that aims to consolidate several forks, that project could themselves publish modules suitable for for replacing those forks. That might look something like:

replace (
	github.com/form3tech-oss/jwt-go => github.com/golang-jwt/jwt/forward/form3tech-oss v0.1.0
	github.com/dgrijalva/jwt-go     => github.com/golang-jwt/jwt/forward/dgrijalva v0.1.0
	github.com/bytewayio/jwt-go     => github.com/golang-jwt/jwt/forward/bytewayio v0.1.0
	github.com/vladimiroff/jwt-go	=> github.com/golang-jwt/jwt/forward/vladimiroff v0.1.0
	github.com/elorusso/jwt-go      => github.com/golang-jwt/jwt/forward/elorusso v0.1.0
)

[bcmills]

So, for another example of package-to-package migration, consider something like golang.org/x/sync/syncmap or golang.org/x/exp/slices, where an experimental or unstable package is proposed for inclusion in some larger project, used as a standalone package during prototyping, and finally adopted in the larger project more-or-less unchanged.

Today we end up leaving behind manually-written forwarding declarations (see https://cs.opensource.google/go/x/sync/+/master:syncmap/go19.go), which aren't too bad for small packages but are much more tedious to write and maintain for larger packages, and are unfortunately prone to documentation skew over time.

[neild]

As another example, if we'd had this feature at the time, we would have used it to rename github.com/golang/protobuf to google.golang.org/protobuf. (And then we'd have had to make the new protobuf API google.golang.org/protobuf/v2, but we probably should have used that name anyway.)

[Linkangyis]

这个提案不错，但是会存在一些潜在问题，假设这个模块需要最低的go版本是1.21，那么如果1.21以下的用户对这个项目进行编译，并不会有异常提示，只会导致编译结果与预期不一致

比如 old模块是fmt.Println(1)
new模块是fmt.Println(2)

低于1.21编译出来的将会是 fmt.Println(1)而不是fmt.Println(2)
比如在这种情况完全可以编译通过

比如以下这种情况

>>>go version
go version go1.20 windows/amd64

go.mod

module Test

go 1.22

main.go

package main

import (
	"Test/TestModule"
	"fmt"
)

func main() {
	Test("Test")
}
func Test(Value interface{}) {
	fmt.Println(Value)
	fmt.Println(testModule.Test())
}

TestModule/Test.go

//go:forward example.com/to/q
package testModule

func Test() map[int]string {
	return make(map[int]string)
}

[bcmills]

TestModule/Test.go would not be allowed to declare a func Test. A given package can have a //go:forward directive or it can have its own declarations (of functions, types, etc.), but not both.

[Linkangyis]

TestModule/Test.go would not be allowed to declare a func Test. A given package can have a //go:forward directive or it can have its own declarations (of functions, types, etc.), but not both.

这样是不是会导致低于支持的版本会返回一系列的未定义，但是不会告诉开发者原因。
低于支持版本的go会出现这个情况
导入了未使用新特性包可以编译
导入了使用新特性的包以后将无法编译

即使在开发者眼中，这个包的github位置从未变过，但是因为其使用了新特性直接导致了项目在低于支持版本的go无法工作

[bcmills]

Yes, in general you must use a Go version ≥ the version required for your dependencies. This design does not change that.

[Linkangyis]

Yes, in general you must use a Go version ≥ the version required for your dependencies. This design does not change that.

那我认为这个提案不错，对于一些场景来说，可以快速的从旧依赖重定向到新依赖，即使他带来一点点问题，但我认为并不影响这个提案的实用性

[iand]

This proposal needs to describe the interaction between retract and forwarding. Can a retracted version contain packages that forward to another version? What happens when forwards are made to a package that is subsequently retracted?

[bcmills]

This proposal needs to describe the interaction between retract and forwarding.

They are completely orthogonal.

Can a retracted version contain packages that forward to another version?

Yes.

What happens when forwards are made to a package that is subsequently retracted?

The user gets a retraction warning, same as if they had any other transitive import of the package from the retracted module.

[iand]

Is the forward used to rewrite the importing code's go.mod? In other words if my go.mod contains example.com/from/p which is forwarded to example.com/to/q does example.com/to/q ever get written to my go.mod such as by running go mod tidy?

If not then this inline forwarding mechanism is going to create new difficulties tracing dependency incompatibilities.

At a minimum the forwarding needs to be exposed by go mod why and similar commands.

[bcmills]

In other words if my go.mod contains example.com/from/p which is forwarded to example.com/to/q does example.com/to/q ever get written to my go.mod such as by running go mod tidy?

Yes. In terms of module dependencies, it should be as if example.com/from/p imports example.com/to/q: go mod tidy should retain both to and from and go mod why should report both p and q.

[matloob]

Thanks for the discussion, this has been very helpful for gathering feedback. Based on the discussion, we're not entirely sure what the right path forward is here, but we'll close this and think some more.