Virtual Environments In Python

Often times, while developing applications in Python, one needs to install some external packages or dependencies.

Normally, third party packages installed using easy_install or pip are typically placed in one of the directories pointed to by the site.getsitepackages() function. For example:

>>> import site
>>> site.getsitepackages()
[
  '/System/Library/Frameworks/Python.framework/Versions/3.5/Extras/lib/python',
  '/Library/Python/3.5/site-packages'
]

By default, every project on your system will use these same directories to store and retrieve site packages (third party libraries). But this becomes a problem if different projects use different versions of the same dependency which Python can’t differntiate between in the site-packages/ directory…. breaking code.

This is where virtual environments come into picture, which provide a way to separate different Python environments for different projects.

Creating a virtual environment

Creating a virtual environment creates a new path for the Python executable as well as a copy of the Python version containing a site-packages folder where each dependency is installed. Activating it sets up your shell to use the environment’s Python executable and its site-packages by default.

Versions of Python 3.6 and above come with the venv module to create virtual environments. For example, running

$ python3 -m venv env

creates a new virtual environment with the name env in the current working directory which would contain all the executable scripts along with the site-packages directory where external dependencies get installed.

For Python2, we can create virtual environments using virtualenv. Before using it, virtualenv should first be installed. We can install using pip like:

$ pip install virtualenv

after which we can create virtual environments like:

$ virtualenv env

To activate the virtual environment run the activate script from the terminal. In *nix systems, from the directory containing the environment variable directory, run:

$ source env/bin/activate
(env) $

This would modify the shell prompt with a prefix like above, indicating the virtual environment is currently active. This means the python executable will only use the environment’s packages and settings, and all external Python packages would be installed in that site-packages directory of the virtual environment.

Now, you can comfortable work on your own isolated Python environment without worrying about breaking code in other projects.

To deactivate the virtual environment, simply run:

(env) $ deactivate
$

This would also remove the prefix indicating the virtual environment has been deactivated.

Managing virtual environments with virtualenvwrapper

As the number of virtual environments in the system increases, it can become difficult to remember and manage them. This is where virtualenvwrapper comes in the picture. It’s just some wrapper scripts around the main virtualenv tool which provides a few features like organizing all virtual environments in one location, commands to create, remove and switch between environments.

To install using pip, run:

$ pip install virtualenv

Once installed, activate its shell functions by running source on the installed virtualenvwrapper.sh script.

For that, first find the location of virtualenvwrapper.sh by running:

$ which virtualenvwrapper.sh
/usr/local/bin/virtualenvwrapper.sh

which should return the path of the virtualenvwrapper.sh script like above.

Using that path, add the following three lines to your shell’s startup file. If you’re using the Bash shell, you would place these lines in either the ~/.bashrc file or the ~/.profile file like so:

export WORKON_HOME=$HOME/.virtualenvs   # Optional
export PROJECT_HOME=$HOME/projects      # Optional
source /usr/local/bin/virtualenvwrapper.sh

and reload the startup file:

$ source ~/.bashrc

There should now be a directory located at $WORKON_HOME that contains all of the virtualenvwrapper data/files.

Now, to create a new virtual environment, simply run:

$ mkvirtualenv env
(env) $

This will create and activate a new environment in the directory located at $WORKON_HOME, where all virtualenvwrapper environments are stored.

If you have many environments to choose from, you can list them all with the workon function:

$ workon
env
env1
env2

And activate a specific virtual environment like:

$ workon env2
(env2) $

Using different Python versions

Sometimes, we want to switch between different versions of Python itself. For that, we have pyenv which we can install in *nix systems with Homebrew like:

$ brew install pyenv

or using pyenv-installer like:

$ curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash

Once installed, we can use it to install specific version of Python, use them for global or local use. Using the pyenv local command, to set the Python version for a specific project or directory by storing the version number in a .local-version file.