Why Are There No Makefiles For Automation In Python Projects?

As a long time Python programmer, I wonder, if a central aspect of Python culture eluded me a long time: What do we do instead of Makefiles? Most ruby-projects I've seen (not just

Solution 1:

Actually, automation is useful to Python developers too!

Invoke is probably the closest tool to what you have in mind, for automation of common repetitive Python tasks: https://github.com/pyinvoke/invoke

With invoke, you can create a tasks.py like this one (borrowed from the invoke docs)

from invoke import run, task

@taskdefclean(docs=False, bytecode=False, extra=''):
    patterns = ['build']
    if docs:
        patterns.append('docs/_build')
    if bytecode:
        patterns.append('**/*.pyc')
    if extra:
        patterns.append(extra)
    for pattern in patterns:
        run("rm -rf %s" % pattern)

@taskdefbuild(docs=False):
    run("python setup.py build")
    if docs:
        run("sphinx-build docs docs/_build")

You can then run the tasks at the command line, for example:

$ invoke clean
$ invoke build --docs

Another option is to simply use a Makefile. For example, a Python project's Makefile could look like this:

docs:$(MAKE) -C docs clean
    $(MAKE) -C docs html
    open docs/_build/html/index.html

release: clean
    python setup.py sdist upload

sdist: clean
    python setup.py sdist
    ls -l dist

Solution 2:

Setuptools can automate a lot of things, and for things that aren't built-in, it's easily extensible.

• To run unittests, you can use the setup.py test command after having added a test_suite argument to the setup() call. (documentation)
• Dependencies (even if not available on PyPI) can be handled by adding a install_requires/extras_require/dependency_links argument to the setup() call. (documentation)
• To create a .deb package, you can use the stdeb module.
• For everything else, you can add custom setup.py commands.

But I agree with S.Lott, most of the tasks you'd wish to automate (except dependencies handling maybe, it's the only one I find really useful) are tasks you don't run everyday, so there wouldn't be any real productivity improvement by automating them.

Solution 3:

There is a number of options for automation in Python. I don't think there is a culture against automation, there is just not one dominant way of doing it. The common denominator is distutils.

The one which is closed to your description is buildout. This is mostly used in the Zope/Plone world.

I myself use a combination of the following: Distribute, pip and Fabric. I am mostly developing using Django that has manage.py for automation commands.

It is also being actively worked on in Python 3.3

Solution 4:

Any decent test tool has a way of running the entire suite in a single command, and nothing is stopping you from using rake, make, or anything else, really.

There is little reason to invent a new way of doing things when existing methods work perfectly well - why re-invent something just because YOU didn't invent it? (NIH).

Solution 5:

The make utility is an optimization tool which reduces the time spent building a software image. The reduction in time is obtained when all of the intermediate materials from a previous build are still available, and only a small change has been made to the inputs (such as source code). In this situation, make is able to perform an "incremental build": rebuild only a subset of the intermediate pieces that are impacted by the change to the inputs.

When a complete build takes place, all that make effectively does is to execute a set of scripting steps. These same steps could just be deposited into a flat script. The -n option of make will in fact print these steps, which makes this possible.

A Makefile isn't "automation"; it's "automation with a view toward optimized incremental rebuilds." Anything scripted with any scripting tool is automation.

So, why would Python project eschew tools like make? Probably because Python projects don't struggle with long build times that they are eager to optimize. And, also, the compilation of a .py to a .pyc file does not have the same web of dependencies like a .c to a .o.

A C source file can #include hundreds of dependent files; a one-character change in any one of these files can mean that the source file must be recompiled. A properly written Makefile will detect when that is or is not the case.

A big C or C++ project without an incremental build system would mean that a developer has to wait hours for an executable image to pop out for testing. Fast, incremental builds are essential.

In the case of Python, probably all you have to worry about is when a .py file is newer than its corresponding .pyc, which can be handled by simple scripting: loop over all the files, and recompile anything newer than its byte code. Moreover, compilation is optional in the first place!

So the reason Python projects tend not to use make is that their need to perform incremental rebuild optimization is low, and they use other tools for automation; tools that are more familiar to Python programmers, like Python itself.