Sharing a Go library to Python (using CFFI)

Disclaimer

I am not a Go expert so I may not be able to answer questions you may have about this process. I am simply trying to reproduce and document what I saw. Additionally, it was actually recommended to not do this for long running processes because you'll have two runtimes that may work against each other eventually (garbage collection, etc). Be wary.

Introduction

Back in July I went to Go Camp that was a part of Open Camps in NYC. It was an informal setting, which actually made it much more comfortable for someone relatively new to the Go programming language.

At the conference Filippo Valsorda (@FiloSottile) did an off the cuff session where he took a Go function and then created a C shared library (.so) out of it (added in Go 1.5). He then took the C shared library and generated a Python shared object that can be directly imported in Python and executed.

Filippo actually has an excellent blog post on this process already. The one difference between his existing blog post and what he presented at Go Camp is that he used CFFI (as opposed to defining CPython extensions) to generate the shared object files that could then be imported directly into Python.

In an attempt to recreate what Filippo did I created a git repo and I'll use that to demonstrate the process that was showed to us at Go Camp. If desired there is also an archive of that git repo here.

System Set Up

I ran this on a Fedora 24 system. To set a Fedora 24 system up from scratch I ran:

$ sudo dnf install golang git python2 python2-cffi python2-devel redhat-rpm-config

I then set my GOPATH. On your system you may already have your GOPATH set:

$ export GOPATH=~/go

Then I cloned the git repo with the example code for this blog post and changed into that directory:

$ go get github.com/dustymabe/go2python-example
$ cd $GOPATH/src/github.com/dustymabe/go2python-example/

Hello From Go

Now we can look at our the file that contains the function we want to export to Python:

$ cd _Go
$ cat hello.go
package main

import "C"
import "fmt"
import "math"

//export Hello
func Hello() {
   fmt.Printf("Hello! The square root of 4 is: %g\n", math.Sqrt(4))
}

func main() {
    Hello()
}

In this file you can see that we are importing a few standard libraries and then defining a function that prints some text to the terminal.

The import "C" is part of cgo and the //export Hello comment right before the function declaration is where we tell go that we want to export the Hello function into the shared library.

Before we generate a shared library, let's test to see if the code compiles and runs:

$ go build -o hello .
$ ./hello
Hello! The square root of 4 is: 2

Now let's generate a shared library:

$ go build -buildmode=c-shared -o hello.so .
$ ls
hello hello.go  hello.h  hello.so  README.md  vendor
$ file hello.so
hello.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ecf1770f0897ca064aab8dacbcb5f7c2f688f34d, not stripped

The hello.so and hello.h files were generated by go. The .so is the shared library that we can now use in C.

Hello From C

We could jump directly to Python now, but we are going to take a detour and just make sure we can get the shared object we just created to work in a simple C program. Here we go:

$ cd ../_C
$ tree .
.
├── hello.c
├── hello.h -> ../_Go/hello.h
├── hello.so -> ../_Go/hello.so
└── README.md

0 directories, 4 files
$
$ cat hello.c
#include "hello.h"

void main() {
     Hello();
}
$ gcc hello.c hello.so
$ LD_LIBRARY_PATH=$(pwd) ./a.out
Hello! The square root of 4 is: 2

What did we just do there? Well, we can see that hello.h and hello.so are symlinked to the files that were just created by the go compiler. Then we show the simple C program that just includes the hello.h header file and calls the Hello() function. We then compile that C program and run it.

We also set the LD_LIBRARY_PATH to $(pwd) so that the runtime shared library loader can find the shared object (hello.so) at runtime and then we ran the program.

So... It worked! Everything looks good in C land.

Hello From Python

For Python we'll first generate the shared object that can be imported directly into Python (just like any .py file). To do this we are using CFFI. A good example that is close to what we are doing here is in the CFFI API Mode documentation.

Here is the file we are using:

$ cd ../_Python/
$ tree .
.
├── hello_ffi_builder.py
├── hello.h -> ../_Go/hello.h
├── hello.py
├── hello.so -> ../_Go/hello.so
└── README.md

0 directories, 5 files
$
$ cat hello_ffi_builder.py
#!/usr/bin/python
from cffi import FFI
ffibuilder = FFI()

ffibuilder.set_source("pyhello",
    """ //passed to the real C compiler
        #include "hello.h"
    """,
    extra_objects=["hello.so"])

ffibuilder.cdef("""
    extern void Hello();
    """)

if __name__ == "__main__":
    ffibuilder.compile(verbose=True)

The ffibuilder.set_source("pyhello",... function sets the name of the file that will get created (pyhello.so) and also defines the code that gets passed to the C compiler. Additionally, it specifies some other objects to load (hello.so). The ffibuilder.cdef defines what program we are building into a shared object; in this case extern void Hello();, so we are just stealing what was defined in hello.so.

Let's run it and see what happens:

$ ./hello_ffi_builder.py
running build_ext
building 'pyhello' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c pyhello.c -o ./pyhello.o
gcc -pthread -shared -Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ./pyhello.o hello.so -L/usr/lib64 -lpython2.7 -o ./pyhello.so
$ ls pyhello.*
pyhello.c  pyhello.o  pyhello.so
$ file pyhello.so
pyhello.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=9a2670b5d287fe80180b158a61ea3e35086e89d7, not stripped

OK. A program (pyhello.c) was generated and then compiled into a shared object (pyhello.so). Can we use it?

Here is the hello.py file that imports the library from pyhello.so and then runs the Hello() function:

$ cat hello.py
#!/usr/bin/python

from pyhello import ffi, lib
lib.Hello()

Does it work?:

$ LD_LIBRARY_PATH=$(pwd) ./hello.py
Hello! The square root of 4 is: 2

You bet!

- Dusty