PyGrunn: embedding the python interpreter - Mark Boer¶

Tags: pygrunn, python

(One of my summaries of a talk at the 2019 PyGrunn conference).

Writing scripts inside applications is often hard. Some of them luckily have an embedded version of python, but not all of them.

Two important terms: extending and embedding. Lots of scientific software is made available via extending: a python wrapper. Numpy and tensorflow, for instance.

The other way around is embedding: you put python inside your application. Useful for plugins, scripting. He doesn’t know if jupyter notebooks are a good example of embedding, but in any case, jupyter is doing funny things with the python interpreter.

CPython, which is the version of python we’re talking about, consists of three parts:

• Bytecode compiler

• Python virtual machine (the one running the compiled bytecode).

• Python’s C API, which allows other programs to call python. The C API is the opposite of python: it is hard to read and write :-) Oh, and the error messages are horrendous.

But… starting python from C and sending lines to the REPL, that’s quite easy. PyRun-SimpleString(). He showed a 10-line C program that reads from stdin and lets python execute it.

He then expanded it to run in a separate thread. But soon his program crashed. The solution was to explicitly acquire and release the GIL (“global interpreter lock”).

A problem: multiprocessing doesn’t work. At least on windows. Starting another process from within python opens another version of the whole program you’re in…

A suggestion: pybind11, a handy library for helping you embed python into c++. It especially helps with managing the GIL and for embedding python modules.

Something he sees often, is that it is used to parallellize code for the benefit of python:

• Convert python types to c/c++ types

• release GIL

• Perform computation

• acquire GIL

• Convert c/c++ types to return type.

A note on deployment: just include python in your installer.