• Creating Python extensions in C

    19Nov
    Categories: development, python Comments: 2

    Hi there!

    In my current work I’m working on optimizing some parallel software, and basically tries to make programs run faster. Within this work there are focus areas such as I/O and memory utilization, which are key areas when trying to optimize software. Generally when people think of highly optimized software, they think of C/C++ and possible assembly.

    Python is very simple in its syntax, and its speed is not at all bad. From a HPC (High Performance Computing) perspective Python may not look that interesting, but combining Python’s simplicity with C’s speed, you’ll get the best from both worlds.

    Since many of the core modules, and Python itself is written in C, it is possible to further extend Python in C. Even the official Python documentation (http://docs.python.org) have a whole section covering the Python C API. By downloading the Python development files (and of course the GNU compiler; apt-get install build-essential), then you’re ready to create C extensions for Python. The overall goal for your C extensions should be to only do the very compute intensive tasks, and keep the other stuff in Python. A key observation here is to identify which parts of your program you may need to create a C extension for – and for this to be found you can use a profiler. Included in Python (“batteries included”) there are a couple of profilers:

    • import profile
    • import cProfile
    • import hotshot

    I will not cover  these python profilers here, but rather inspire you to try them out. They might save you a lot of work and give you valuable knowledge of your software. Do some reading of python profiling here.

    Now lets create a very simple (and extremely stupid) python extension in C. We want it to contain two methods:

    1. a method which allows you to run shell commands by passing a string
    2. a method which returns some text based on your input

    To create this extension we follow these steps: a) decide the python module name b) create the C file with the same name as the python module name c) start programming C! Let’s name our module for “cool” (not the best name)..

    This is our cool.c file:

    #include <stdlib.h>
#include <stdio.h>
#include <Python.h>
 
/*
 * Takes a string argument, which is the shell command, and runs it.
 * Returns the return code of the system(cmd) call.
 * */
static PyObject *cool_command(PyObject *self, PyObject *args) {
    const char *cmd;
    int retval;
 
    if(!PyArg_ParseTuple(args, "s", &cmd))
        return NULL;
 
    retval = system( cmd );
    return Py_BuildValue("i", retval);
}
 
/*
 * Takes a string argument, and assembles it into a new python string
 * object and returns it.
 * */
static PyObject *cool_greet(PyObject *self, PyObject *args) {
    char *input;
    char *resp = "Hi there: ";
 
    if(!PyArg_ParseTuple(args, "s", &input))
        return NULL;
 
    char *retstr = (char *) malloc( sizeof(char)* (strlen(input)+strlen(resp)) + 1 ); // +1 &lt;- null termination
    if (retstr == NULL)
        return NULL;
 
    PyObject *retString;
 
    strcpy( retstr, resp );
    strcat(retstr, input);
 
    retString = Py_BuildValue("s", retstr);
    Py_INCREF(retString);
    free(retstr);
 
    return retString;
}
 
/* an array of PyMethodDef structure. A PyMethodDef structure is
used to describe a method of an extension type. */
static PyMethodDef CoolMethods[] = {
    {"command", cool_command, METH_VARARGS, "Execute a shell command." },
    {"greet", cool_greet, METH_VARARGS, "Returns a greeting to the caller."},
    {NULL, NULL, 0, NULL}
};
 
PyMODINIT_FUNC initcool(void)
{
    (void) Py_InitModule( "cool", CoolMethods );
}

    This C extension file is built up with the following key sections:

    • Inclusion of headers.
    • static PyObject methods which constitutes the visible python functions of our extension module.
    • An array of PyMethodDef structures. A PyMethodDef structure describes a method of an extension type.
    • A initialization method which is named “init<modulename>“. This method has the sole purpose of calling the Py_InitModule function which takes the name and the PyMethodDef array.

    So there you go, this is our Python extension implemented in C with the help of the Python development headers. Now we’ll have a look at our Distutils script (setup.py) that will assist us with the compilation and creation of the actual python module.

    This is our setup.py file:

    from distutils.core import setup, Extension
 
module1 = Extension('cool',
        sources = ['cool.c'] )
 
setup ( name = 'CoolPackage',
        version = '0.1',
        description = 'A descriptive and informal C extension to Python',
        ext_modules = [module1] )

    jada..

    This file is much self-explained. We have basically one method named “setup” which takes a bunch of arguments. These arguments is the package name, version, description and a set of Extension objects. These Extension objects describes our C extension files. By simply running this command:

    python setup.py build

    When the compilation is done, you’ll have a build folder in the same directory as the setup.py. Go into the “build/lib.linux-i686-2.6/“. Then type “python” or “ipython”, and then “import cool”. Now you actually have loaded the “cool” module, and you may call “cool.greet(’Alex’)” and/or “cool.command(’ls /’)” and the actual computing happens in the C world instead of the Python world.

    Now, keep in mind that this C extension isn’t actual doing anything useful. But, given that you have some algorithm or other problem to solve, and the time is of the essence, then utilizing the power which lays in this C extensions can give you significant time savings.

    Tags: , , , ,

  • Discovering python’s functools.partial powers

    10Nov
    Categories: Uncategorized Comments: 1

    Hi there.

    I just recently discovered the power of the functional function named partial inside the python module “functools”. Functools provides: “Tools for working with functions and callable objects”, and comes with python v2.5 and up.

    But, let’s illustrate some of the beauty with examples.

    import functools
def check_balance(amount, limit=0):
    if amount &lt;= limit:
        return False
    return True
positive_account_balance = functools.partial( check_balance, limit=1 ) # new func object to check if account is more than 0
rich_dude = functools.partial( check_balance, limit=1000 ) # new func object to check if this person has more than 1000 in his account
poor_dude = functools.partial( check_balance, limit=-100 ) # new func object to check if this person is poor (i.e. less than 100)

    So, let me explain some of this code. We are first defining a function named “check_balance” which takes two arguments (amount and limit). This function is very simple, as it only checks if amount is less than or equal to limit and returns a boolean value.

    So, what really does the partial function do? It returns a new function object based on the function object passed in as the first argument, and the predefined arguments provided. In other words we construct new function objects where some or all of the arguments is given, and then call that particular function later on, thus saving us from writing the function arguments again.

    Another example could be that you want a function to check if a username is also and administrator:

    import functools
groups = {'admin': ['asbjorn'], 'users': ['asbjorn', 'lolcat', 'frank']}
is_admin = functools.partial( lambda user, group: user in groups.get(group), group="admin" )
is_admin("asbjorn") # will return True

    This functionality makes it very simple to create detailed and specific functions based on more generic designed functions. This also makes it simpler to follow the design approach Domain-driven Design (DDD) and  create a Domain Specific Language, as its easier to create domain specific functions.

Blogroll:

Meta:

Recent Posts