After Python scripts are mixed into a project, where should dependencies be installed? venv and PEP 668 in practice

This article starts from a real error report and discusses not how to fix this one issue, but rather Boundaries of Python dependencies in multilingual projects How should it be rowed. Regardless of whether your project is Node, Go, Rust, or pure frontend, as long as [something] gets mixed in .py Script, this applies to this entire article.

I. The scene of the incident: two commands, two errors

There is a Python check script in the project (scripts/*.py), during local development they are chained together via npm script. One day, after typing the start command, the first link failed:

> python3 scripts/source-language-docs-check-colon.py

Traceback (most recent call last):
  File ".../scripts/source-language-docs-check-colon.py", line 5, in <module>
    import yaml
ModuleNotFoundError: No module named 'yaml'

My first reaction was to just pretend and it would be fine:

python3 -m pip install pyyaml

As a result, it hit the second error:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.

    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:

    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz

    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with

    brew install pipx

    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.

    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.

    Read more about this behavior here: <https://peps.python.org/pep-0668/>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

a simple import, it was blocked by Python at two points. Let us break down these two barriers below.

II. The First Pitfall:yaml Not Python standard library

See import yaml, intuitively one would think it is a built-in module, but it is not. It is not in the Python standard library yaml.

Here is yaml From third-party package PyYAML:

• Installation name:pyyaml

• Import name:yaml

The name not matching is a pitfall many people encounter when installing it for the first time. So the correct approach is indeed pip install pyyaml. The real issue is not whether to pretend, but rather Which environment should it be installed in?.

III. Second Pitfall: PEP 668 and [externally managed environment]

externally-managed-environment This error comes from PEP 668. Its meaning is:

This Python environment is managed by the system (such as Homebrew, Debian/Ubuntu apt). Please do not install third-party packages into it, to avoid breaking the system Python.

This is actually Protection mechanism In the past, many people used sudo pip install Breaking the system Python and getting stuck in a cycle where upgrading breaks it and not upgrading also breaks it. PEP 668 essentially seals off the system Python. The error message itself also lists all the possible ways out:

• Path A:brew install xyz, applicable to libraries already packaged by Homebrew

• Path B:python3 -m venv ..., applicable to any third-party library

• Path C:pipx install xyz, suitable for command-line tools

• Path D:--break-system-packages, emergency exit, at your own risk

IV. Core Question: Where Exactly Should Dependencies Be Installed?

When Python scripts are mixed into a project, there are four installation approaches before you, each with its own pros and cons:

Installation method	Command	Applicable scenarios	Risk
System Python	sudo pip install	Do not use	Pollute the system, crash on upgrade
User level	pip install --user	Small tools on a personal machine	Cross-project version conflict
Emergency exit	--break-system-packages	Temporary emergency assistance	Sabotage Homebrew management
Project venv	python3 -m venv .venv	Project-level dependency	Almost none

Conclusion: project dependencies go in the project venv. This is the only isolated, rebuildable, reproducible option.

cd /path/to/your-project

# 1. Create a virtual environment
python3 -m venv .venv

# 2. Activate
source .venv/bin/activate

# 3. Install dependencies (make sure the venv is activated)
pip install pyyaml

# 4. Verify
python3 -c "import yaml; print(yaml.__version__)"

After success, print the version number (e.g.6.0.3) is then complete.

V. Why Not Use --break-system-packages

The easiest approach is obviously python3 -m pip install --break-system-packages pyyaml, one line gets it done, the error disappears. But do not make it the first choice, reasons:

• Breaking the Homebrew management contract: Future brew upgrade python Possible conflict.

• Delayed outbreak of problems Just because no error is reported today, it does not mean none will be reported six months later. Mutual interference among library versions across projects is the hardest type of bug to investigate.

• Irreproducible: Switch to another machine, or switch to a different colleague, and the environment is different.

The cost of venv is just three commands plus one line .gitignore. Trade [convenience] for [rebuildability]; with project-level dependencies, you never lose.

Six. A detail: why no need to change the script

your package.json(or Makefile, shell scripts) are usually bare python3:

{
  "scripts": {
    "check": "python3 scripts/some-check.py"
  }
}

After venv is activated,python3 can Automatically resolved to .venv/bin/python3, rely on PATH priority to handle it. So:

• No need to change what is in the script.python3 path

• No need to set PYTHONPATH

• Just [activate first, then run]

source .venv/bin/activate
npm run docs:dev   # or make dev, or any command that invokes python3

7. Wrap-up: Consolidate the Experience

Fixing once is luck; turning the fix into a standard is an asset. Three things are worth solidifying into the repository:

1..gitignore Ignore venv

.venv/

2.requirements.txt Fixed version

Warning

If the project does not yet have requirements.txt File, please create it first:

touch requirements.txt

Then fill in:

pyyaml==6.0.3

3. Add [First-time Configuration] to README

### Prerequisites (Python)

Some scripts in this project use Python, so a virtual environment needs to be created:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

Optional:make dev One-click solution

dev:
	@if [ ! -d .venv ]; then python3 -m venv .venv && .venv/bin/pip install -r requirements.txt; fi
	@. .venv/bin/activate && npm run docs:dev

the next sentence make dev, if venv does not exist it will be created automatically and dependencies installed, so newcomers can clone it with zero thinking.

8. Five Lessons to Take Away

• import yaml Not the standard library. It comes from PyYAML; the package name and the import name are not the same, so do not be fooled by the name.

• PEP 668 is a good thing.externally-managed-environment It is protecting the system Python. When you encounter it, you should be grateful rather than try to bypass it.

• --break-system-packages is the last resort It exists, but it should not be the first reaction.

• venv cost is extremely low. Run the command three times and add one line..gitignore, and then the problem of [where to package] can be permanently solved.

• It only counts as done once the writing is finished.. Turn the pitfalls into documentation, README, or make target is the closing action with the highest team benefit.

One-sentence summary: See again ModuleNotFoundError: No module named 'xxx', do not rush yet pip install, first ask a question [Which project does this dependency belong to?]. The answer is usually the venv of that project.