The Ultimate Solution to go:embed Not Supporting Symlinks and Missing Hidden Files

Abstract: When developing a Go scaffolding tool, have you ever encountered //go:embed Problems such as files disappearing out of thin air after packaging, hidden files being unreadable, and symlinks failing? This article documents the pitfalls encountered and proposes a brand-level solution based on the idea of dimensionality reduction and dimensional elevation.

1. Crime scene: The 3 files that vanished into thin air

Recently, I have been developing a Go project scaffolding generation tool (apply framework), I encountered an extremely bizarre Bug.

I used a synchronization tool locally to extract the project template, and the console explicitly indicated that it had generated 394 template files. However, when users use our compiled CLI tool to initialize a project, the generated directory contains only 391 a file.

3 files are missing!

After searching for a long time, I finally discovered that these 3 missing files are all in the master.Soft link (Symlink). For example: docker/debian/etc/config.yaml is a symbolic link pointing to ../../../cmd/apiserver/etc/config.yaml.

The files are sitting perfectly fine in the source directory—so why do they disappear at runtime after being packaged into a Go binary?

2. Getting to the Bottom of It: The Hidden Rules of go:embed

The problem lies with the Go language.//go:embed in the underlying design of instructions. In Go code, we embed the template directory into the binary:

//go:embed embeds
var embeds embed.FS

but //go:embed There are two very frustrating “hidden rules” that usually won’t raise an error:

• Ignore by default . and _ files beginning with If you put something in the template directory .gitignore or .env File, directly embed It cannot be read; it will automatically filter it out for you.

• The Symlink Disaster When go:embed When encountering a symbolic link, its behavior is very unpredictable. If it attempts to dereference (Dereference), and the target file does not exist or is outside the module directory, it will Directly discard this file silently..

This is why our binary is missing those 3 symlinks.The Go compiler threw them away during packaging.

3. Step 1 of fixing the pit: force inclusion and full bundling.

To solve the problem of missing hidden files, we must give go:embed add all: Prefix.

//go:embed all:embeds
var embeds embed.FS

Add all: After that, hidden files can indeed be packaged in. However,The physical nature of a symbolic link still cannot be brought into the binary..

We can't stuff a real Unix symlink into a binary file. Forcing dereference would break the original project structure.

4. Ultimate dimensionality reduction strike:__APPLY_SYMLINK__ Plan

Since it cannot be preserved at the physical level, we will solve it at the logical level:Record through dimensionality reduction, reconstruct through dimensionality elevation..

To solve this problem, we designed the following mechanism (and added the framework's brand prefix):

Step 1: Dimensionality reduction (when extracting templates)

Our sync tool (applygenctl) When traversing the master, if a symbolic link is found, no longer attempt to generate a real physical symbolic link, but instead Generate a plain text file.(bring along .tmpl suffix).

The content of the file is merely a single line containing a brand-prefixed identifier and the target path:

__APPLY_SYMLINK__:../../../cmd/apiserver/etc/config.yaml

Step 2: Dimensional elevation (when generating the project)

When the user's command-line tool (applyctl) When it reads the embedded template, it checks the file contents. If it finds that the first line begins with __APPLY_SYMLINK__: At the beginning, it will immediately understand:

“Oh! This is a symbolic link, not a regular file!”

At this point, the tool strips the prefix, extracts the actual relative path, and in the user's target directory, through a system call os.Symlink Recreate this physical symbolic link.

Step 3: Perfect restoration

When we once again pass through tree When viewing the generated project structure:

docker
└── debian
    ├── builder.Dockerfile
    ├── etc
    │   └── config.yaml -> ../../../cmd/apiserver/etc/config.yaml
    └── service -> ../../supervisor/linux

Everything is exactly the same as the master template! The cross-platform Go project file generation challenge has been perfectly solved.

5. Summary and Recommendations

If you are also using Go to develop scaffolding, code generators, or any project that needs to package static resources, please remember the following two points:

• Always use //go:embed all:xxx, unless you are 100% sure you don't need hidden files.

• Don't believe it go:embed Handling of symbolic links and special files. If the file system structure needs to be preserved, use “Metadata dimensionality reduction” (e.g., record prefix __APPLY_SYMLINK__) is the only reliable cross-platform solution.

This pitfall not only allowed us to fill in go:embed vulnerabilities, and incidentally left the framework with highly recognizable code characteristics. Free things are often the most expensive; you only truly master the pitfalls of the open-source ecosystem after stepping in them.