Hi there!
In my current work I’m working on optimizing some parallel software, and basically tries to make programs run faster. Within this work there are focus areas such as I/O and memory utilization, which are key areas when trying to optimize software. Generally when people think of highly optimized software, they think of C/C++ and possible assembly.
Python is very simple in its syntax, and its speed is not at all bad. From a HPC (High Performance Computing) perspective Python may not look that interesting, but combining Python’s simplicity with C’s speed, you’ll get the best from both worlds.
Since many of the core modules, and Python itself is written in C, it is possible to further extend Python in C. Even the official Python documentation (http://docs.python.org) have a whole section covering the Python C API. By downloading the Python development files (and of course the GNU compiler; apt-get install build-essential), then you’re ready to create C extensions for Python. The overall goal for your C extensions should be to only do the very compute intensive tasks, and keep the other stuff in Python. A key observation here is to identify which parts of your program you may need to create a C extension for – and for this to be found you can use a profiler. Included in Python (“batteries included”) there are a couple of profilers:
- import profile
- import cProfile
- import hotshot
I will not cover these python profilers here, but rather inspire you to try them out. They might save you a lot of work and give you valuable knowledge of your software. Do some reading of python profiling here.
Now lets create a very simple (and extremely stupid) python extension in C. We want it to contain two methods:
- a method which allows you to run shell commands by passing a string
- a method which returns some text based on your input
To create this extension we follow these steps: a) decide the python module name b) create the C file with the same name as the python module name c) start programming C! Let’s name our module for “cool” (not the best name)..
This is our cool.c file:
#include <stdlib.h> #include <stdio.h> #include <Python.h> /* * Takes a string argument, which is the shell command, and runs it. * Returns the return code of the system(cmd) call. * */ static PyObject *cool_command(PyObject *self, PyObject *args) { const char *cmd; int retval; if(!PyArg_ParseTuple(args, "s", &cmd)) return NULL; retval = system( cmd ); return Py_BuildValue("i", retval); } /* * Takes a string argument, and assembles it into a new python string * object and returns it. * */ static PyObject *cool_greet(PyObject *self, PyObject *args) { char *input; char *resp = "Hi there: "; if(!PyArg_ParseTuple(args, "s", &input)) return NULL; char *retstr = (char *) malloc( sizeof(char)* (strlen(input)+strlen(resp)) + 1 ); // +1 <- null termination if (retstr == NULL) return NULL; PyObject *retString; strcpy( retstr, resp ); strcat(retstr, input); retString = Py_BuildValue("s", retstr); Py_INCREF(retString); free(retstr); return retString; } /* an array of PyMethodDef structure. A PyMethodDef structure is used to describe a method of an extension type. */ static PyMethodDef CoolMethods[] = { {"command", cool_command, METH_VARARGS, "Execute a shell command." }, {"greet", cool_greet, METH_VARARGS, "Returns a greeting to the caller."}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initcool(void) { (void) Py_InitModule( "cool", CoolMethods ); }
This C extension file is built up with the following key sections:
- Inclusion of headers.
- static PyObject methods which constitutes the visible python functions of our extension module.
- An array of PyMethodDef structures. A PyMethodDef structure describes a method of an extension type.
- A initialization method which is named “init<modulename>“. This method has the sole purpose of calling the Py_InitModule function which takes the name and the PyMethodDef array.
So there you go, this is our Python extension implemented in C with the help of the Python development headers. Now we’ll have a look at our Distutils script (setup.py) that will assist us with the compilation and creation of the actual python module.
This is our setup.py file:
from distutils.core import setup, Extension module1 = Extension('cool', sources = ['cool.c'] ) setup ( name = 'CoolPackage', version = '0.1', description = 'A descriptive and informal C extension to Python', ext_modules = [module1] )
jada..
This file is much self-explained. We have basically one method named “setup” which takes a bunch of arguments. These arguments is the package name, version, description and a set of Extension objects. These Extension objects describes our C extension files. By simply running this command:
python setup.py build
When the compilation is done, you’ll have a build folder in the same directory as the setup.py. Go into the “build/lib.linux-i686-2.6/“. Then type “python” or “ipython”, and then “import cool”. Now you actually have loaded the “cool” module, and you may call “cool.greet(’Alex’)” and/or “cool.command(’ls /’)” and the actual computing happens in the C world instead of the Python world.
Now, keep in mind that this C extension isn’t actual doing anything useful. But, given that you have some algorithm or other problem to solve, and the time is of the essence, then utilizing the power which lays in this C extensions can give you significant time savings.
Have a good summer people!