Dynamic vs. static vs. shared

It's rarely known, that Singular has a kind of plug-in mechanism. We call it dynamic modules and it uses dlopen to add some extended kernel functionality to Singular, while keeping the binary executable small. (It might be better called shared module - we will see below.)

The module itself is compiled and linked like a shared library. Well, maybe the linker options are a little bit unusual. I realized this, while preparing the pyobject extension. The latter enables Singular to handle objects from Python, in particular it enables Singular to load and execute routines from our sister-project PolyBoRi.

I finished the extension itself some weeks ago, but in the first it was statically linked into the Singular binary. Hence, my personal development copy of the Singular binary got dependent  on libpython. To avoid this the Singular-team came up with the idea to make a dynamic module out of the extension. The principle change was not too complicated, since I already gained some experience with the dynamic module stuff, while preparing pyobject's predecessor psico. The extension is still deactivated per default, but it can easily be activated while configuring:

./configure --with-python

After rebuilding (make install) a Singular session can seamlessly access PolyBoRi.
> python_import("polybori");
> def r = declare_ring(list(Block("x", 10), Block("y", 10)));
> list polybori_ideal = (x(1)+x(2),x(2)+y(1));
> def result = groebner_basis(polybori_ideal);
> result;
[x(1) + y(1), x(2) + y(1)]
> Auf Wiedersehen.

After that, things got complicated and the real work was starting.  My pyobject.so (the plug-in) depended on the runtime library libpython (and the dependencies of the latter). In general, this is not bad: if you build Singular from scratch, building succeeds if and only if the dependencies were there in the first place. It is also not bad, if a package management system (rpm, deb and consorts) does the building, because it resolves such dependencies for you. To cut a long speech short: if you want to use Python, you'll have to install Python on your system. By the way, I don't know any full-featured distribution (Linux or Unix), which runs without it, so this is not really a challenge. (It would be if you want to install Singular on an embedded system. But that's a challenge anyway.)

On the other hand the Singular team distributes fall-back binaries for those users which are not able to build Singular on their own. These binaries must not have any external dependencies, because system libraries vary from distro to distro.

What about the dynamic modules which already come with Singular?
Answer:
> ldd p_Procs_FieldQ.so
        statically linked
This was a surprise. Since dynamic modules should make the binary small, I did not expect the modules itself to be linked statically, because - incorporating a whole bunch of system library stuff - now the modules become quite fat.

To be honest, until I typed that ldd command above, I thought, that shared and static are mutually exclusive. Indeed, the opposite of static is dynamic. Albeit the opposite of a shared library is a static one, a shared library itself could be linked dynamically or statically with the system libraries. So statically linked dynamic modules do make sense in the Singular context of providing fall-back binaries.

How can I archive this with pyobject.so? Just adding -shared -Xlinker -static to the Makefile did not do the job. The module was compiled and linked without error, but loading yielded an dlopen error: The symbol _Py_NoneStruct was not found.

A quick google search implied that there is something special with PyNone, so I avoided to use it. But then, the next unresolved symbol occurred. Ah, do'h! dlopen just complains about the first missing symbol. Actually, I never linked to libpython, because I put the libraries in the wrong order and (using -static) the order does matter. Also, linking like this does not complain about missing symbols, because the symbols might be resolved on runtime from the binary or other shared libraries. The latter was forbidden by the -static flag, and so, and so, and so, my module got corrupted.

Finally, I ended up with the following linkage call

  g++ pyobject.dl_o -Xlinker -static -nodefaultlibs -Xlinker -export-dynamic -shared -L/usr/lib64/python2.6 -L/usr/lib64 -lpython2.6  -lpthread -ldl  -lutil -lc -o pyobject.so

where -lpthread -ldl -lutil fulfill the dependencies of libpython2.6 and the construct -nodefaultlibs ... -lc avoids trying to link against some unused shared c++-libs (which - of course - is not possible statically).

Note that this only works, if all libraries are available in their static lib***.a variants. Even more: to get position-independent code (i. e. shared-library conforming) each of which has to be compiled with -fPIC, which is not common for .a-libs. So ./configure --with-python  tests for all of this and falls back to the classical dynamic-shared-modules in case of failure. Also, the user can force to build these light-weight modules by typing
  ./configure --with-python=module,dynamic
at the command prompt. In addition
   ./configure --with-python=embed
will incorporate the pyobject extension completely into the binary.

My best,
  Alexander
P.S.: I'm aware, there are more dependencies besides libpython, for instance the Python standard library (written in Python itself), which needs to be bundled for providing a completely distro-independent binary distribution. But that's another story.