Go toolchain automatic upgrade pitfalls: why go mod tidy Will it secretly download a new Go version?

A legacy project with k8s 1.12.2 go mod tidy Complete troubleshooting of the failure: Go 1.21+'s toolchain automatic upgrade mechanism, and how it bypasses your replace, and GOTOOLCHAIN=local Why can it be solved.

1. Conclusion first

If what you are maintaining is a An old Go project with locked dependencies(k8s, etcd, docker, and the like), and go.mod There’s a pile inside.replace, then:

• Never run it directly go mod tidy, it will trigger an automatic toolchain upgrade and leave the dependency graph in disarray.

• Forever Plus GOTOOLCHAIN=local:

GOTOOLCHAIN=local go mod tidy

• Or, in go.mod hardcoded in:

go 1.26.3
toolchain go1.26.3

This one mnemonic can save you half a day.

Below is the complete process of the pitfalls encountered, the underlying principles, and the solutions.

II. Phenomenon: Two commands, one succeeds and the other fails

I am maintaining an old CSI driver project from the k8s 1.12.2 era. One day I ran:

go mod tidy

Go printed a line of log I didn't pay much attention to:

go: downloading go1.27.1 (darwin/amd64)
go: toolchain upgrade needed to resolve k8s.io/[email protected]

Then,It downloaded Go 1.27.1 by itself., and starts throwing a bunch of errors:

module k8s.io/apimachinery@latest found (v0.37.0), but does not contain package k8s.io/apimachinery/pkg/util/clock
module k8s.io/client-go@latest found (v0.37.0), but does not contain package k8s.io/client-go/util/integer
google.golang.org/genproto/googleapis/api/expr/v1alpha1: ambiguous import: found package ... in multiple modules

pkg/util/clock,util/integer These packages clearly existed in k8s 1.12.2 in 2018, so how could they "not exist"?

What's even stranger is that I tried it:

GOTOOLCHAIN=local go mod tidy

It succeeded—there wasn't a single error.

the same one go mod tidy Should I add it or not?GOTOOLCHAIN=local, and the results are vastly different.

III. Principle: Go 1.21+ toolchain automatic upgrade mechanism

3.1 go.mod inside go Instructions are not just language versions.

go 1.26.3

This line has two meanings:

• Declare the Go language version used by the project..

• Declare the minimum toolchain version required by the project.

If a certain dependency's go.mod wrote in it go 1.27, Go will think that what you declared 1.26.3 Not enough.

3.2 Default Behavior GOTOOLCHAIN=auto

Go 1.21 introduced an environment variable.GOTOOLCHAIN, the default value is auto. Its semantics is:

If the current toolchain does not meet the dependency requirements,Automatically download and use a higher toolchain.

Therefore go mod tidy See k8s.io/[email protected]... Requirement go >= 1.27.0, then:

• Download go1.27.1.

• Switch to this new toolchain.

• Reparse the entire dependency graph under the new toolchain..

3.3 Why “Reparsing” Causes Problems

The key lies in:The parsing rules used by the new toolchain, and what you originally wrote replace the rules used may be inconsistent.

• You are at go.mod write in:

replace k8s.io/apimachinery => k8s.io/apimachinery v0.0.0-20181022183627-f71dbbc36e12

• but go1.27.1 During parsing, it may be due to the dependency graph's Other branches, go @latest take v0.37.0, then overwrite your replace.

• What is even more concealed is:k8s.io/kubernetes v1.12.2 some subpath of it imports a package, and this package exists in 1.12.2, but go1.27.1 go @latest taken k8s.io/apimachinery v0.37.0 inside No.

• So you see pkg/util/clock does not exist This kind of "inexplicable" error.

essence: toolchain auto-upgrade =Use the parsing rules of the new Go version to rerun the dependency graph of the old project.. And your replace It was written for the old rules and may become invalid under the new rules.

IV. Why GOTOOLCHAIN=local Can be saved

GOTOOLCHAIN=local The semantics are:

Do not download any other toolchain; just use the local one. If requirements are not met, report an error; do not make decisions on your own.

Therefore:

• Does not trigger toolchain switching.

• Use go1.26.3 of parsing rules.

• Honestly, according to you.go.mod written inside replace Go.

• go mod tidy Success.

One sentence:GOTOOLCHAIN=local Make Go behave and not change the toolchain or dependency versions on its own.

Five.GOTOOLCHAIN the four values of

value	behavior
auto(Default)	Allow automatic download and use of a higher version of the toolchain
local	Force use of the local toolchain; do not upgrade.
go1.27.1	Specify a specific toolchain
go1.27.1+auto	Allow further upgrades based on the specified version.

For legacy projects,Always use local.

VI. Three Pitfall-Prevention Solutions

Option A: In go.mod hardcode the toolchain in

go 1.26.3
toolchain go1.26.3

toolchain The guild mandates the use of a specified version.

Note:go instructions and toolchain Instructions are two different things.

• go 1.26.3: Language version + minimum toolchain requirements.

• toolchain go1.26.3: The actual toolchain version used.

go mod tidy Only look go Instruction, not looking toolchain So if go written too low, high dependency requirements,It will still trigger automatic toolchain upgrades., regardless toolchain Did you write it or not?

The correct approach is Write both., or directly use Plan B.

Option B: Global environment variables (most recommended)

go env -w GOTOOLCHAIN=local

After that, all Go commands will not automatically upgrade. Set it once and it takes effect permanently.

Option C: Upgrade go.mod of go Instruction to 1.27

go 1.27

Make the dependency requirements satisfied without triggering an upgrade. However, it will change the language version and may have side effects.

Recommend A or B, because legacy projects are more controllable.

7. Under what circumstances would one fall into this pitfall?

• The project depends on old modules from 2018–2022 (k8s, etcd, docker, etc.).

• of these old modules go.mod Not written go Instruction (or written low), but Indirect dependency There are new modules inside that require a higher version.

• go.mod There are many in it.replace Let's lock the version.

• used @latest or a similar parsing method.

As long as you hit 2 or more of them, you're very likely to fall into the trap.

VIII. Troubleshooting Checklist

When you run into pitfalls, troubleshoot in the following order:

• go env GOTOOLCHAIN Look at the current settings.

• go.mod Is there any inside?toolchain Okay.

• Are there any in the dependencies?go 1.27+ of the module.

• replace Whether it has been bypassed.

• Add GOTOOLCHAIN=local Retry.

Nine, my final go.mod What does it look like?

After a full round of troubleshooting, my go.mod Here, the k8s-related dependencies look like this:

require (
	k8s.io/api v0.0.0-20181026184759-d1dc89ebaebe // indirect
	k8s.io/apimachinery v0.0.0-20181022183627-f71dbbc36e12 // indirect
	k8s.io/apiserver v0.0.0-20181026185746-f1e867e1a455 // indirect
	k8s.io/client-go v0.0.0-20181026185218-bf181536cb4d // indirect
	k8s.io/csi-api v0.0.0-20181026191722-d3fde979a63c // indirect
	k8s.io/kube-openapi v0.0.0-20180711000925-0cf8f7e6ed1d // indirect
	k8s.io/utils v0.0.0-20180726175726-66066c83e385 // indirect
	k8s.io/kubernetes v1.12.2
)

replace (
	bitbucket.org/ww/goautoneg => github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822
	google.golang.org/genproto => google.golang.org/genproto v0.0.0-20231120223509-83a465c0220f
	k8s.io/api => k8s.io/api v0.0.0-20181026184759-d1dc89ebaebe
	k8s.io/apiextensions-apiserver => k8s.io/apiextensions-apiserver v0.0.0-20181026191334-ba848ee89ca3
	k8s.io/apimachinery => k8s.io/apimachinery v0.0.0-20181022183627-f71dbbc36e12
	k8s.io/apiserver => k8s.io/apiserver v0.0.0-20181026185746-f1e867e1a455
	k8s.io/client-go => k8s.io/client-go v0.0.0-20181026185218-bf181536cb4d
	k8s.io/csi-api => k8s.io/csi-api v0.0.0-20181026191722-d3fde979a63c
	k8s.io/kube-openapi => k8s.io/kube-openapi v0.0.0-20180711000925-0cf8f7e6ed1d
	k8s.io/utils => k8s.io/utils v0.0.0-20180726175726-66066c83e385
)

Key points:

• all k8s.io/* All submodules are replace to the 2018 pseudo-version.

• k8s.io/apiextensions-apiserver must At the same time appear in require and replace in (replace Only replace, do not introduce).

• bitbucket.org/ww/goautoneg Use a GitHub mirror instead.

• google.golang.org/genproto Lock to the 2023-11 version to avoid conflicts with submodules googleapis/rpc Conflict.

10. Conclusion

The official Go toolchain automatic upgrade is goodwill: If a dependency requires a higher version, rather than reporting an error, it is better to automatically download a new version to run.

But this in An old project with very strict dependency version locking. Neighborhood meeting Do more harm than good by trying to help:

• It automatically downloads Go 1.27.1.

• Under the new toolchain, some @latest Parsing was re-triggered.

• So those in the old project that were replace The pinned old version, by the parsing rules of the new toolchain Bypass.

A one-sentence mnemonic:

Legacy projects, locked dependencies, there are replace,go Add before the command.GOTOOLCHAIN=local.

Appendix: Environment for the pitfalls in this article

• Go 1.26.3 (local machine)

• Project:github.com/example/csi-driver, dependency k8s.io/kubernetes v1.12.2

• Auto-upgrade trigger version:go1.27.1

• Original error message:

module k8s.io/apimachinery@latest found (v0.37.0), but does not contain package k8s.io/apimachinery/pkg/util/clock
google.golang.org/genproto/googleapis/api/expr/v1alpha1: ambiguous import

If you've also fallen into this trap, feel free to leave yours in the comments.go.mod and the error—I'll help you determine whether it was caused by an automatic toolchain upgrade or some other issue.

Reference

• Go official toolchain documentation:

• Go 1.21 Release Notes:

• GOTOOLCHAIN Environment variable:

Note: This article is based on actual testing with Go 1.26.3; behavior may change in future versions.