Introduction to virtual environments

Jeffrey M. Perkel, The sleight-of-hand trick that can simplify scientific computing, Nature 617, 212-213 (2023).

https://docs.python.org/3/tutorial/venv.html

https://packaging.python.org/tutorials/installing-packages/

https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/

https://virtualenv.pypa.io/en/stable/user_guide.html

RATIONALE

Python applications will often use packages and modules that don’t come as part of the standard library. Applications will sometimes need a specific version of a library, because the application may require that a particular bug has been fixed or the application may be written using an obsolete version of the library’s interface.

This means it may not be possible for one Python installation to meet the requirements of every application. The solution for this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.

Currently, there are two common tools for creating Python virtual environments:

'venv' is available by default in Python 3.3 and later, and installs 'pip' and 'setuptools' into created virtual environments in Python 3.4 and later.
'virtualenv' needs to be installed separately, but supports Python 2.7+ and Python 3.3+, and 'pip', 'setuptools' and 'wheel' are always installed into created virtual environments by default (regardless of Python version).

It is useful to have a separate directory for virtual environments, e.g. VIRTUAL in the home directory.

Warning. Created python virtual environments are usually not self-contained. A complete python packaging is usually made up of thousands of files, so it’s not efficient to install the entire python again into a new folder. Instead virtual environments are mere shells, that contain little within themselves, and borrow most from the system python. This does mean that if you upgrade your system python your virtual environments might break.

# Using 'venv' in Linux.

$ python3 -m venv DIR
# DIR can be replaced with ENV_ML for some machine learning project
$ source DIR/bin/activate
# some actions ...
(DIR) $ deactivate

# Using 'venv' in Windows.

python -m venv DIR
DIR\Scripts\activate
# some actions ...
deactivate

# Using 'virtualenv' in Linux.

$ virtualenv DIR
$ source DIR/bin/activate
# some actions ...
(DIR) $ deactivate

# Using 'virtualenv' in Windows.

virtualenv DIR   # the full path may be needed
DIR\Scripts\activate
# some actions ...
deactivate

In order to remove a virtual environment you should simply deactivate it and remove DIR directory (rm -r DIR). You might want to pip freeze a dependency list first.

PYTHON SCRIPTS IN VIRTUAL ENVIRONMENTS

#!/usr/bin/env python   # or 'python3' if you have Py2 and Py3
#
# hello.py

print("Hello!")

(DIR) $ python hello.py     # it is safe for Py2 and Py3
Hello!
(DIR) $ chmod a+x hello.py
(DIR) $ ./hello.py     # problems with '#!/usr/bin/python'
Hello!
(DIR) $