It's rarely known, that Singular has a kind of plug-in mechanism. We call it
dynamic modules and it uses
dlopen to add some extended kernel functionality to Singular, while keeping the binary executable small. (It might be better called
shared module - we will see below.)
The module itself is compiled and linked like a shared library. Well, maybe the linker options are a little bit unusual. I realized this, while preparing the
pyobject extension. The latter enables Singular to handle objects from Python, in particular it enables Singular to load and execute routines from our sister-project
PolyBoRi.
I finished the extension itself some weeks ago, but in the first it was statically linked into the Singular binary. Hence, my personal development copy of the Singular binary got dependent on libpython. To avoid this the Singular-team came up with the idea to make a dynamic module out of the extension. The principle change was not too complicated, since I already gained some experience with the dynamic module stuff, while preparing pyobject's predecessor
psico. The extension is still deactivated per default, but it can easily be activated while configuring:
./configure --with-python
After rebuilding (
make install) a Singular session can seamlessly access PolyBoRi.
> python_import("polybori");
> def r = declare_ring(list(Block("x", 10), Block("y", 10)));
> list polybori_ideal = (x(1)+x(2),x(2)+y(1));
> def result = groebner_basis(polybori_ideal);
> result;
[x(1) + y(1), x(2) + y(1)]
> Auf Wiedersehen.
After that, things got complicated and the real work was starting. My
pyobject.so (the plug-in) depended on the runtime library libpython (and the dependencies of the latter). In general, this is not bad: if you build Singular from scratch, building succeeds if and only if the dependencies were there in the first place. It is also not bad, if a package management system (rpm, deb and consorts) does the building, because it resolves such dependencies for you. To cut a long speech short: if you want to use Python, you'll have to install Python on your system. By the way, I don't know any full-featured distribution (Linux or Unix), which runs without it, so this is not really a challenge. (It would be if you want to install Singular on an embedded system. But that's a challenge anyway.)
On the other hand the Singular team distributes fall-back binaries for those users which are not able to build Singular on their own. These binaries must not have any external dependencies, because system libraries vary from distro to distro.
What about the dynamic modules which already come with Singular?
Answer:
> ldd p_Procs_FieldQ.so
statically linked
This was a surprise. Since dynamic modules should make the binary small, I did not expect the modules itself to be linked statically, because - incorporating a whole bunch of system library stuff - now the modules become quite fat.
To be honest, until I typed that
ldd command above, I thought, that shared and static are mutually exclusive. Indeed, the opposite of
static is
dynamic. Albeit the opposite of a shared library is a static one, a shared library itself could be linked dynamically or statically with the system libraries. So
statically linked dynamic modules do make sense in the Singular context of providing fall-back binaries.
How can I archive this with
pyobject.so? Just adding
-shared -Xlinker -static to the Makefile did not do the job. The module was compiled and linked without error, but loading yielded an dlopen error: The symbol
_Py_NoneStruct was not found.
A quick google search implied that there is something special with
PyNone, so I avoided to use it. But then, the next unresolved symbol occurred. Ah, do'h! dlopen just complains about the
first missing symbol. Actually, I never linked to libpython, because I put the libraries in the wrong order and (using
-static) the order does matter. Also, linking like this
does not complain about missing symbols, because the symbols might be resolved on runtime from the binary or other shared libraries. The latter was forbidden by the -static flag, and so, and so, and so, my module got corrupted.
Finally, I ended up with the following linkage call
g++ pyobject.dl_o -Xlinker -static -nodefaultlibs -Xlinker -export-dynamic -shared -L/usr/lib64/python2.6 -L/usr/lib64 -lpython2.6 -lpthread -ldl -lutil -lc -o pyobject.so
where
-lpthread -ldl -lutil fulfill the dependencies of libpython2.6 and
the construct -nodefaultlibs ... -lc avoids trying to link against some unused shared c++-libs (which - of course - is not possible statically).
Note that this only works, if all libraries are available in their static
lib***.a variants. Even more: to get position-independent code (i. e. shared-library conforming) each of which has to be compiled with -fPIC, which is not common for
.a-libs. So .
/configure --with-python tests for all of this and falls back to the classical dynamic-shared-modules in case of failure. Also, the user can force to build these light-weight modules by typing
.
/configure --with-python=module,dynamic
at the command prompt. In addition
.
/configure --with-python=embed
will incorporate the pyobject extension completely into the binary.
My best,
Alexander
P.S.: I'm aware, there are more dependencies besides libpython, for instance the Python standard library (written in Python itself), which needs to be bundled for providing a completely distro-independent binary distribution. But that's another story.
