What's the difference between dist-packages and site-packages?

dist-packages is a Debian-specific convention that is also present in its derivatives, like Ubuntu. Modules are installed to dist-packages when they come from the Debian package manager into this location:

/usr/lib/python2.7/dist-packages

Since easy_install and pip are installed from the package manager, they also use dist-packages, but they put packages here:

/usr/local/lib/python2.7/dist-packages

From the Debian Python Wiki:

dist-packages instead of site-packages. Third party Python software installed from Debian packages goes into dist-packages, not site-packages. This is to reduce conflict between the system Python, and any from-source Python build you might install manually.

This means that if you manually compile and install Python interpreter from source, it uses the site-packages directory. This allows you to keep the two installations separate, especially since Debian and Ubuntu rely on the system version of Python for many system utilities.

Debian (and Ubuntu) has introduced its own convention

# python3 -m site

on Ubuntu Focal gives

sys.path = [
    '/qpid-dispatch',
    '/usr/lib/python38.zip',
    '/usr/lib/python3.8',
    '/usr/lib/python3.8/lib-dynload',
    '/usr/local/lib/python3.8/dist-packages',
    '/usr/lib/python3/dist-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.8/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

The convention, as described in the linked mailing list, is that python deb packages installed by the distribution package manager go into /usr/lib/python3/dist-packages and packages installed using sudo pip3 go into /usr/local/lib/python3.8/dist-packages.

If you compile and install your own Python interpreter, it will default to placing itself in /usr/local, with /usr/local/bin/pip3 installs going into /usr/local/lib/pythonX.Y/site-packages.

The point of the Debian convention is to keep the three sets of packages separate:

1. python packages installed by apt
2. packages installed by root user with /usr/bin/pip3
3. packages installed by root user with their own /usr/local/bin/pip3

dist-packages is the debian-specific directory where apt and friends install their stuff, and site-packages is the standard pip directory.

The problem is -- what happens when different versions of the same package are present in different directories?

My solution to the problem is to make dist-packages a symlink to site-packages:

for d in $(find $WORKON_HOME -type d -name dist-packages); do
  pushd $d
  cd ..
  if test -d dist-packages/__pycache__; then
    mv -v dist-packages/__pycache__/* site-packages/__pycache__/
    rmdir -v dist-packages/__pycache__
  fi
  mv -v dist-packages/* site-packages/
  rmdir -v dist-packages
  ln -sv site-packages dist-packages
  popd
done

(if you are not using gnu tools, remove the -v option).