Mastering Python Virtual Environments: From venv to pip-tools

In the world of Python development, managing dependencies and project-specific environments is a foundational skill. It ensures that your projects are isolated, reproducible, and free from conflicts. For years, the standard has been venv combined with a requirements.txt file. In this post, we’ll revisit this classic approach and then explore how pip-tools can significantly improve the workflow.

The Classic: venv and requirements.txt

The venv module, included in Python 3, allows us to create lightweight virtual environments. Each environment has its own Python interpreter and installed packages, isolated from other projects and the system-wide Python installation.

The workflow is straightforward:

1. Create an environment:

   python3 -m venv venv

2. Activate it:

   • On macOS/Linux: source venv/bin/activate
   • On Windows: .\venv\Scripts\activate

3. Install packages:

   pip install requests
   pip install flask

4. Freeze dependencies:

   pip freeze > requirements.txt

This last step creates a requirements.txt file, which contains a list of all the packages installed in the environment, including their exact versions and all their sub-dependencies. While this ensures reproducibility, it has a major drawback: it doesn’t distinguish between the packages you directly need (like requests) and the packages they depend on (like charset-normalizer, idna, urllib3, and certifi).

This makes the requirements.txt file difficult to manage. If you want to update a direct dependency, you have to manually update it and then re-freeze, hoping you don’t break anything. If you want to remove a dependency, you have to manually clean up its sub-dependencies.

A Better Way: pip-tools

This is where pip-tools comes in. It introduces a more declarative and manageable approach to dependencies with two key files: requirements.in and requirements.txt.

The pip-tools Workflow:

1. Install pip-tools:

   pip install pip-tools

2. Define your direct dependencies:
   Create a file named requirements.in and list only the packages your project directly depends on. You can specify versions if you need to.

   # requirements.in
   flask
   requests

3. Compile your dependencies:
   Run the pip-compile command:

   pip-compile requirements.in

This command takes your requirements.in file, resolves all the necessary sub-dependencies, and generates a comprehensive requirements.txt file. This file is beautifully commented, showing which top-level package each sub-dependency belongs to.

#
# This file is autogenerated by pip-compile with Python 3.9
# To update, run:
#
#    pip-compile requirements.in
#
# autogenerated from requirements.in
#
flask==2.0.2
    # via -r requirements.in
requests==2.27.1
    # via -r requirements.in
...

4. Sync your environment:
   Instead of pip install -r requirements.txt, you use pip-sync:

   pip-sync

   The pip-sync command ensures that your virtual environment has exactly the packages listed in requirements.txt. It will install missing packages, update incorrect versions, and even remove packages that are not part of the requirements.

Conclusion

By separating your direct dependencies (requirements.in) from the full, locked dependency list (requirements.txt), pip-tools provides a much cleaner and more maintainable workflow. It makes updating packages safer and keeps your project’s dependency tree transparent and easy to understand. If you’re still using pip freeze, give pip-tools a try—it’s a simple change that can make a big difference.