Tuesday, June 12, 2012

In the Aftermath of the ELAGB Workshop

Last week we've had the first Efficient Linear Algebra for Gröbner Basis Computations workshop. Located here in Fraunhofer ITWM we had found the most inspiring conference environment here in Kaiserslautern. Combining lectures and coding sprint sessions the workshop structure was highly influenced by the Sage Days concept. But - rather than supporting a specific system - our coding projects focussed  on exact linear algebra methods for computing Gröbner bases, in particular those related to the F4 algorithm.
The atmosphere was terrific: computer algebra icons lecturing by day and infinite coding all night long! Martin Albrecht gives quite a few impression about our worksome week in his blog. Also you find the full lectures at YouTube. Friday approached soon, too soon! We are looking forward for another workshop of this kind!

Sunday, February 19, 2012

Hacking a circular dependency away

This is a horror story on how messy a SW design may be.
Particularly sensitive people are advised not to read on! ;-)

Hello guys,



this week I finally managed to enable Super Commutative Algebras (SCA) for spielwiese. And that’s with the approximately 1 year old fix of the general non-commutative (NC) stuff... and even at that time it was more of a hack. So what’s the matter with NC and SCA?


Recall that in the legacy Singular all this stuff was packed together into the core, therefore it was kind of Ok to have circular dependencies between subsystems, e.g. between NC and GB (Groebner Bases).


During the refactoring Singular effort, NC went to libpoly since it provides the low-level multiplication and other basic routines, whereas GB stayed at libkernel. The point is that they (unfortunately) depend on each other in the following way:

1. InitNC (an entry part of NC) is widely called in libpolys

2. InitSCA (called by InitNC) needs NF (from GB)

3. InitNC and InitSCA set a ring member as a pointer to a function like gnc_GB/sca_GB

4. the later functions (*_GB) and NF are implemented within the GB framework and absolutelycannot be separated from it,

5. clearly GB is based on libpolys (for doing arithmetic) and thus implicitly use basic NC functions


Since separating libpolys (containing polynomial arithmetic basics) from libkernel (containing higher level algorithms) was among of our goals for refactoring Singular we cannot simply pull GB down to libpolys. Thus we have got the following picture: on the very bottom libpolys widely uses InitNC, which needs *_GB and NF, which are defined on higher level in libkernel within GB, which in turn depends on libpolys. Thus: lower layer depends on higher and of course other way around, i.e. a circular dependency between layers!


In order to make this design to work with pure brute force we declare all needed functions (NF, *_GB) in libpolys. Of course, its test suite cannot link now due to undefined symbols.
Thus we add dummy definitions to each test unit. Yey! This makes the test suite pass, since we luckily don’t test any NC stuff there :-/


That must be all to it!? Wat! What’s that? Now we cannot link the static test suite of libkernel! And liner complains that *_GB functions are missing! How so? Well the main point is that that those *_GB function are _never_ called in libkernel itself! Hence our clever linker discards them while linking the static test units against libkernel and miss them while linking against libpolys, as we, of course, pass “-lkernel -lpolys” to it.


Unfortunately, forcefully adding libpolys to libkernel (with something like libkernel_LIBADD = libpolys.la) did not do the job :(


Well, the other usual trick in this situation is to duplicate the library usage specification and make linker to consider it twice (remember - we are linking a static binary!). Great - using “-lkernel -lpolys -lkernel” actually works from command line, and if we proceed we can also link Singular static binaries by hand. Moreover, this trick will have to be done whenever linking against both libkernel and libpolys.


Oh, well, anyway... Are we done? Not quite :’-( “Luckily” we use autotools & libtool - so we start changing those LDFLAGS/LDADD for test units, and... Hey!? Hello again? As it turns out our libtool removes library duplicates... The solution is easy, we add “LIBTOOLFLAGS = --preserve-dup-deps” to our Makefile.am, and... Hey!?? Hello again?? Well, as it turns out libtool is clever enough to detect and remove these “unneeded” library duplicates even with that option!!!


Now we are desperate enough to try to forcefully pass to linker something like --start-group/--end-group or --whole-archive/--no-whole-archive. Alas, no luck!


THIS IS THE END OF THE WORLD!!!!... of NC/SCA-Singular!?



Nope. The solution is right under our nose: we may fool the linker into believing that those functions are used by libkernel using many tricks. My way was the following: recall that the *_GB functions were needed for setting a function pointer (for C-style of virtual method overloading). That function pointer will be used by corresponding wrapper NCGB by GB and it must not be NULL it the NC-ring was initialized properly... this wrapper must not be very efficient so we can test for NULL pointer and kind of explain the logic behind InitNC/InitSCA with actual calls to gnc_GB and sca_GB in a never visited if-branch.


Yeah! Unbelievable, that finally did the job!


The main drawback is that whoever links against libpolys (but not against libkernel) has to define (in the main object unit) several bizarre function. Imho, that is ugly! Although, this can be simplified up to including an additional special-purpose header, which does just that.


I can see some alternatives:

1. One “simple” alternative design solution would be to separate “the setting a ring member as a pointer to a function like gnc_GB/sca_GB” artificially from NC and move to GB. Well, i guess this might lead to even bigger mess...

2. A large scale alternative solution would be to separate NC (any maybe the calling parts) from libpolys, GB from libkernel and put them together as a new library in between. Yeah libpolys will still depend on InitNC and other NC high-level functions but one could turn them all into global function pointers... which are initially NULL and are to be set on higher level. Well, imho, it’s not a nice solution but might work.

3. Rewrite the inherited C-like code in C++ using inheritance for NC-struct and virual methods for slow initialization methods.

As a conclusion I'd say that maintaining old and tricky C-like C++ code provides new tricky to solve problems the further it goes... It is worth the trouble to rewrite it in correct C++!

Tuesday, January 10, 2012

C vs. C++'s C

Lately, compiling some cygwin-based Windows port failed here: 
char buffer[255];
snprintf(buffer, 255, "%s.png", name);

By taking into account buffer's length the snprintf command yields a more secure variant of sprintf introduced by the C99 standard.

It turned out that the problem - or course! - was not cygwin-related, but due to a slightly newer version of gcc which did not know about snprintf. Prepending the following lines of codes
#ifndef snprintf
#define snprintf(buffer, len, arg1, arg2) \
  sprintf(buffer, arg1, arg2)
#endif
fixed the problem, but that was far from being sophisticated. (Remember: we loose the important buffer-length information!)

So I started wondering why gcc had stopped supporting a long-time introduced and useful standard-conforming command. Also, enforcing standard by the -std=c99 command line option didn't help.

Of course, it didn't! Indeed, the problem was caused by the selected language dialect, but our code above was part of some C++ code. It was compiled with -std=c++98 to conform with the established C++ standard - approved one year before C99! Hence, the C-part of C++98 differed due to some last minute changes in C.

So what? Should I completely skip the nice snprintf for portability reasons? Yes, but not by using the error prone sprintf.

The real solution was to stay in (proper) C++ all the time:
std::ostringstream fullname;
fullname << name << ".png";
const char* buffer =  fullname.str().c_str();

Final philosophy:
Coding C++ code code C++ code correctly.

My best,
  Alexander

Tuesday, September 6, 2011

SICSA2011

Since 29.Aug till 1.Sep there was a small summer school at St.Andrews, Scotland: Advanced techniques in computer algebra systems development.

It was about CAS internals, memory management/garbage collection, parallelization, thread-safety etc.

In particular, here is my talk about the Singular memory management library: omalloc.

Wednesday, March 16, 2011

Testing contributed libraries

A new Singular release is in the pipeline. In the new release, some libraries distributed as experimental previously will be updated and promoted to standard. We wanted to send an email to experimental library authors in order to remind them to
  • send us the latest version of their libraries and
  • check if the library conforms to the requirements for standard libraries.
Apart from the examples provided in the documentation, each library should provide a list of commands and the corresponding output to check the correctness of all code paths. These will be included in the Singular test suite to verify that the library works as intended on all platforms supported by Singular in future releases.

While working on the email, it became apparent that the instructions for adding tests for libraries are buried in the source code. For easy access before they are published in the online manual for the next release, here are the lines from the file Tst/README with instructions on providing test data.

To add a new test for a library/command to the test-suite:
  • For most library files, two sets of test files need to be provided.
    • Short tests should test the essential functionality of the library/command in a relatively short time (say, in no more than 30s).
    • Long tests should test the functionality of the library/command in detail so that, if possible, all relevant cases/results are tested. Nevertheless, such a test should not run longer than, say, 10 minutes.
    If useful tests generally execute in a short time, it is enough to have short tests only.
  • Each set should include
    • a tst file containing the code to execute the test (details of the format of these files are explained below),
    • and a res file with the corresponding output.
  • The test files should have the following name convention (replace xx with library name):
    • xx_s.tst: Singular code for short and basic tests
    • xx_s.res: Output of xx_s.tst
    • xx_l.tst: Singular code for long and extended tests 
    • xx_l.res: Output of xx_l.res
    or, alternatively:
    • xx.tst: Singular code for short tests, only
    • xx.res: Output of xx.tst
Rules for providing tst files:
  1. tst files always start with the following three commands as preamble:
    LIB "tst.lib";
    tst_init();
    tst_ignore("CVS ID $Id$"); // or version number here
    tst_init() writes some general info to stdout (like date,  uname, hostname, version, etc.). The library tst.lib (contained in the Singular distribution) provides, among others, the routines tst_init() and tst_ignore().
  2. tst files should end with the following statements:
    tst_status(1); $
    which enables (automatic) checks of the timing/memory performance of Singular.
  3. All system-dependent output (like run-times, memory usages, pathnames, dates, etc.) should generally be avoided.
  4. After time/memory critical sections of the tst files, the command
    tst_status();
    should be inserted. This enables (automatic) checks of the  timing/memory performance of Singular since the last call to tst_status() (resp. since the start-up of Singular).
  5.  If system-dependent output can not be avodied, the routine tst_ignore() should be used:
    tst_ignore(val [, keyword]): 'val' can have arbitrary type for which a string conversion exists; if present, keyword must be one of the following strings: "time", "memory"
    tst_ignore() outputs 'val' by adding the following prefix:
    no keyword -- // tst_ignore:
    "time" keyword -- // tst_ignore: time:
    "memory" keyword -- // tst_ignore: memory:
    which causes automatic tests to ignore these lines when doing a diff on result files. 
An example of a short tst file can be found here:
http://www.singular.uni-kl.de:8002/trac/browser/trunk/Tst/Short/gmssin_s.tst

You can download the newest version of tst.lib from
ftp://www.mathematik.uni-kl.de/pub/Math/Singular/singular-cd/LIBRARIES/tst.lib


Monday, February 14, 2011

Dynamic vs. static vs. shared

It's rarely known, that Singular has a kind of plug-in mechanism. We call it dynamic modules and it uses dlopen to add some extended kernel functionality to Singular, while keeping the binary executable small. (It might be better called shared module - we will see below.)

The module itself is compiled and linked like a shared library. Well, maybe the linker options are a little bit unusual. I realized this, while preparing the pyobject extension. The latter enables Singular to handle objects from Python, in particular it enables Singular to load and execute routines from our sister-project PolyBoRi.

I finished the extension itself some weeks ago, but in the first it was statically linked into the Singular binary. Hence, my personal development copy of the Singular binary got dependent  on libpython. To avoid this the Singular-team came up with the idea to make a dynamic module out of the extension. The principle change was not too complicated, since I already gained some experience with the dynamic module stuff, while preparing pyobject's predecessor psico. The extension is still deactivated per default, but it can easily be activated while configuring:
./configure --with-python

After rebuilding (make install) a Singular session can seamlessly access PolyBoRi.
> python_import("polybori");
> def r = declare_ring(list(Block("x", 10), Block("y", 10)));
> list polybori_ideal = (x(1)+x(2),x(2)+y(1));
> def result = groebner_basis(polybori_ideal);
> result;
[x(1) + y(1), x(2) + y(1)]
> Auf Wiedersehen.


After that, things got complicated and the real work was starting.  My pyobject.so (the plug-in) depended on the runtime library libpython (and the dependencies of the latter). In general, this is not bad: if you build Singular from scratch, building succeeds if and only if the dependencies were there in the first place. It is also not bad, if a package management system (rpm, deb and consorts) does the building, because it resolves such dependencies for you. To cut a long speech short: if you want to use Python, you'll have to install Python on your system. By the way, I don't know any full-featured distribution (Linux or Unix), which runs without it, so this is not really a challenge. (It would be if you want to install Singular on an embedded system. But that's a challenge anyway.)

On the other hand the Singular team distributes fall-back binaries for those users which are not able to build Singular on their own. These binaries must not have any external dependencies, because system libraries vary from distro to distro.

What about the dynamic modules which already come with Singular?
Answer:
> ldd p_Procs_FieldQ.so
        statically linked
This was a surprise. Since dynamic modules should make the binary small, I did not expect the modules itself to be linked statically, because - incorporating a whole bunch of system library stuff - now the modules become quite fat.

To be honest, until I typed that ldd command above, I thought, that shared and static are mutually exclusive. Indeed, the opposite of static is dynamic. Albeit the opposite of a shared library is a static one, a shared library itself could be linked dynamically or statically with the system libraries. So statically linked dynamic modules do make sense in the Singular context of providing fall-back binaries.

How can I archive this with pyobject.so? Just adding -shared -Xlinker -static to the Makefile did not do the job. The module was compiled and linked without error, but loading yielded an dlopen error: The symbol _Py_NoneStruct was not found.

A quick google search implied that there is something special with PyNone, so I avoided to use it. But then, the next unresolved symbol occurred. Ah, do'h! dlopen just complains about the first missing symbol. Actually, I never linked to libpython, because I put the libraries in the wrong order and (using -static) the order does matter. Also, linking like this does not complain about missing symbols, because the symbols might be resolved on runtime from the binary or other shared libraries. The latter was forbidden by the -static flag, and so, and so, and so, my module got corrupted.

Finally, I ended up with the following linkage call
  g++ pyobject.dl_o -Xlinker -static -nodefaultlibs -Xlinker -export-dynamic -shared -L/usr/lib64/python2.6 -L/usr/lib64 -lpython2.6  -lpthread -ldl  -lutil -lc -o pyobject.so

where -lpthread -ldl -lutil fulfill the dependencies of libpython2.6 and the construct -nodefaultlibs ... -lc avoids trying to link against some unused shared c++-libs (which - of course - is not possible statically).

Note that this only works, if all libraries are available in their static lib***.a variants. Even more: to get position-independent code (i. e. shared-library conforming) each of which has to be compiled with -fPIC, which is not common for .a-libs. So ./configure --with-python  tests for all of this and falls back to the classical dynamic-shared-modules in case of failure. Also, the user can force to build these light-weight modules by typing
  ./configure --with-python=module,dynamic
at the command prompt. In addition
   ./configure --with-python=embed
will incorporate the pyobject extension completely into the binary.

My best,
  Alexander
P.S.: I'm aware, there are more dependencies besides libpython, for instance the Python standard library (written in Python itself), which needs to be bundled for providing a completely distro-independent binary distribution. But that's another story.

Thursday, December 2, 2010

Singular memory

As you probably know: Singular uses omalloc by Olaf Bachmann. It allows fast allocation of memory blocks of the same size.

Just a while ago we released new Singular: 3-1-2. Among other stuff it has the following experimental (and undocumented) feature: one can print out omalloc status (every single memory-related number) AND will try to test omalloc consistency.

In order to call it - simply type:
system("omMemoryTest");


Here follows its output on my machine:

> system("omMemoryTest");

[om_Info]:
MaxBytesSystem : 794624
CurrentBytesSystem : 794624
MaxBytesSbrk : 270336
CurrentBytesSbrk : 270336
MaxBytesMmap : 524288
CurrentBytesMmap : 524288
UsedBytes : 156976
AvailBytes : 493880
UsedBytesMalloc : 126568
AvailBytesMalloc : 0
MaxBytesFromMalloc : 126568
CurrentBytesFromMalloc : 126568
MaxBytesFromValloc : 524288
CurrentBytesFromValloc : 524288
UsedBytesFromValloc : 30408
AvailBytesFromValloc : 493880
MaxPages : 24
UsedPages : 23
AvailPages : 105
MaxRegionsAlloc : 1
CurrentRegionsAlloc : 1

[om_Opts]:
MinTrack : 0
MinCheck : 0
MaxTrack : 5
MaxCheck : 10
Keep : 100
HowToReportErrors : 2
MarkAsStatic : 0
PagesPerRegion : 128
OutOfMemoryFunc : 0x54e510
MemoryLowFunc : (nil)
ErrorHook : 0x877110


[om_ErrorStatus] : 'no error' (omError_NoError)
[om_InternalErrorStatus]: 'no error' (omError_NoError)


The later two lines indicate that no error was found.

PURPOSE: calling this function after a suspicious kernel function might indicate a memory corruption in it.