Run modules as scripts

Since I posted this question, I've learnt that you can run a module as if it were a script using Python's -m command-line switch (which I had thought only applied to packages).

So I think the best solution is this:

• Instead of writing wrapper scripts in a bin subdirectory, put the bulk of the logic in modules (as you should anyway), and put at the end of relevant modules if __name__ == "__main__": main(), as you would in a script.
• To run the scripts on the command line, call the modules directly like this: python -m pkg_name.module_name
• If you have a setup.py, as Alik said you can generate wrapper scripts at installation time so your users don't need to run them in this funny way.

PyCharm doesn't support running modules in this way (see this request). However, you can just run modules (and also scripts in bin) like normal because PyCharm automatically adds the project root to the PYTHONPATH, so import statements resolve without any further effort. There are a few gotchas for this though:

• The main problem the working directory will be incorrect, so opening data files won't work. Unfortunately there is no quick fix; the first time you run each script, you must then stop it and change its configured working directory (see this link).
• If your package directory is not directly within the root project directory, you need to mark its parent directory as a source directory in the project structure settings page.
• Relative imports don't work i.e. you can do from pkg_name.other_module import fn but not from .other_module import fn. Relative imports are usually poor style anyway, but they're useful for unit tests.
• If a module has a circular dependency and you run it directly, it will end up being imported twice (once as pkg_name.module_name and once as __main__). But you shouldn't have circular dependencies anyway.

Bonus command line fun:

• If you still want to put some scripts in bin/ you can call them with python -m bin.scriptname (but in Python 2 you'll need to put an __init__.py in the bin directory).
• You can even run the overall package, if it has a __main__.py, like this: python -m pkg_name

Pip editable mode

There is an alternative for the command line, which is not as simple, but still worth knowing about:

• Use pip's editable mode, documented here
• To use it, make a setup.py, and use the following command to install the package into your virtual environment: pip install -e .
• Note the trailing dot, which refers to the current directory.
• This puts the scripts generated from your setup.py in your virtual environment's bin directory, and links to your package source code so you can edit and debug it without re-running pip.
• When you're done, you can run pip uninstall pkg_name
• This is similar to setup.py's develop command, but uninstallation seems to work better.

The simplest way is to use setuptools in your setup.py script, and use the entry_points keyword, see the documentation of Automatic Script Creation.

In more detail: you create a setup.py that looks like this

from setuptools import setup

setup(
    # other arguments here...
    entry_points={
        'console_scripts': [
            'foo = my_package.some_module:main_func',
            'bar = other_module:some_func',
        ],
        'gui_scripts': [
            'baz = my_package_gui:start_func',
        ]
    }
)

then create other Python packages and modules underneath the directory where this setup.py exists, e.g. following the above example:

.
├── my_package
│   ├── __init__.py
│   └── some_module.py
├── my_package_gui
│   └── __init__.py
├── other_module.py
└── setup.py

and then run

$ python setup.py install

or

$ python setup.py develop

Either way, new Python scripts (executable scripts without the .py suffix) are created for you that point to the entry points you have described in setup.py. Usually, they are at the Python interpreter's notion of "directory where executable binaries should be", which is usually on your PATH already. If you are using a virtual env, then virtualenv tricks the Python interpreter into thinking this directory is bin/ under wherever you have defined that the virtualenv should be. Following the example above, in a virtualenv, running the previous commands should result in:

bin
├── bar
├── baz
└── foo