Since 29.Aug till 1.Sep there was a small summer school at St.Andrews, Scotland: Advanced techniques in computer algebra systems development.
It was about CAS internals, memory management/garbage collection, parallelization, thread-safety etc.
In particular, here is my talk about the Singular memory management library: omalloc.
Showing posts with label Singular. Show all posts
Showing posts with label Singular. Show all posts
Tuesday, September 6, 2011
Monday, February 14, 2011
Dynamic vs. static vs. shared
It's rarely known, that Singular has a kind of plug-in mechanism. We call it dynamic modules and it uses dlopen to add some extended kernel functionality to Singular, while keeping the binary executable small. (It might be better called shared module - we will see below.)
The module itself is compiled and linked like a shared library. Well, maybe the linker options are a little bit unusual. I realized this, while preparing the pyobject extension. The latter enables Singular to handle objects from Python, in particular it enables Singular to load and execute routines from our sister-project PolyBoRi.
I finished the extension itself some weeks ago, but in the first it was statically linked into the Singular binary. Hence, my personal development copy of the Singular binary got dependent on libpython. To avoid this the Singular-team came up with the idea to make a dynamic module out of the extension. The principle change was not too complicated, since I already gained some experience with the dynamic module stuff, while preparing pyobject's predecessor psico. The extension is still deactivated per default, but it can easily be activated while configuring:
After rebuilding (make install) a Singular session can seamlessly access PolyBoRi.
> python_import("polybori");
> def r = declare_ring(list(Block("x", 10), Block("y", 10)));
> list polybori_ideal = (x(1)+x(2),x(2)+y(1));
> def result = groebner_basis(polybori_ideal);
> result;
[x(1) + y(1), x(2) + y(1)]
> Auf Wiedersehen.
After that, things got complicated and the real work was starting. My pyobject.so (the plug-in) depended on the runtime library libpython (and the dependencies of the latter). In general, this is not bad: if you build Singular from scratch, building succeeds if and only if the dependencies were there in the first place. It is also not bad, if a package management system (rpm, deb and consorts) does the building, because it resolves such dependencies for you. To cut a long speech short: if you want to use Python, you'll have to install Python on your system. By the way, I don't know any full-featured distribution (Linux or Unix), which runs without it, so this is not really a challenge. (It would be if you want to install Singular on an embedded system. But that's a challenge anyway.)
On the other hand the Singular team distributes fall-back binaries for those users which are not able to build Singular on their own. These binaries must not have any external dependencies, because system libraries vary from distro to distro.
What about the dynamic modules which already come with Singular?
Answer:
> ldd p_Procs_FieldQ.so
statically linked
This was a surprise. Since dynamic modules should make the binary small, I did not expect the modules itself to be linked statically, because - incorporating a whole bunch of system library stuff - now the modules become quite fat.
To be honest, until I typed that ldd command above, I thought, that shared and static are mutually exclusive. Indeed, the opposite of static is dynamic. Albeit the opposite of a shared library is a static one, a shared library itself could be linked dynamically or statically with the system libraries. So statically linked dynamic modules do make sense in the Singular context of providing fall-back binaries.
How can I archive this with pyobject.so? Just adding -shared -Xlinker -static to the Makefile did not do the job. The module was compiled and linked without error, but loading yielded an dlopen error: The symbol _Py_NoneStruct was not found.
A quick google search implied that there is something special with PyNone, so I avoided to use it. But then, the next unresolved symbol occurred. Ah, do'h! dlopen just complains about the first missing symbol. Actually, I never linked to libpython, because I put the libraries in the wrong order and (using -static) the order does matter. Also, linking like this does not complain about missing symbols, because the symbols might be resolved on runtime from the binary or other shared libraries. The latter was forbidden by the -static flag, and so, and so, and so, my module got corrupted.
where -lpthread -ldl -lutil fulfill the dependencies of libpython2.6 and the construct -nodefaultlibs ... -lc avoids trying to link against some unused shared c++-libs (which - of course - is not possible statically).
Note that this only works, if all libraries are available in their static lib***.a variants. Even more: to get position-independent code (i. e. shared-library conforming) each of which has to be compiled with -fPIC, which is not common for .a-libs. So ./configure --with-python tests for all of this and falls back to the classical dynamic-shared-modules in case of failure. Also, the user can force to build these light-weight modules by typing
./configure --with-python=module,dynamic
at the command prompt. In addition
./configure --with-python=embed
will incorporate the pyobject extension completely into the binary.
My best,
Alexander
P.S.: I'm aware, there are more dependencies besides libpython, for instance the Python standard library (written in Python itself), which needs to be bundled for providing a completely distro-independent binary distribution. But that's another story.
The module itself is compiled and linked like a shared library. Well, maybe the linker options are a little bit unusual. I realized this, while preparing the pyobject extension. The latter enables Singular to handle objects from Python, in particular it enables Singular to load and execute routines from our sister-project PolyBoRi.
I finished the extension itself some weeks ago, but in the first it was statically linked into the Singular binary. Hence, my personal development copy of the Singular binary got dependent on libpython. To avoid this the Singular-team came up with the idea to make a dynamic module out of the extension. The principle change was not too complicated, since I already gained some experience with the dynamic module stuff, while preparing pyobject's predecessor psico. The extension is still deactivated per default, but it can easily be activated while configuring:
./configure --with-python
After rebuilding (make install) a Singular session can seamlessly access PolyBoRi.
> python_import("polybori");
> def r = declare_ring(list(Block("x", 10), Block("y", 10)));
> list polybori_ideal = (x(1)+x(2),x(2)+y(1));
> def result = groebner_basis(polybori_ideal);
> result;
[x(1) + y(1), x(2) + y(1)]
> Auf Wiedersehen.
After that, things got complicated and the real work was starting. My pyobject.so (the plug-in) depended on the runtime library libpython (and the dependencies of the latter). In general, this is not bad: if you build Singular from scratch, building succeeds if and only if the dependencies were there in the first place. It is also not bad, if a package management system (rpm, deb and consorts) does the building, because it resolves such dependencies for you. To cut a long speech short: if you want to use Python, you'll have to install Python on your system. By the way, I don't know any full-featured distribution (Linux or Unix), which runs without it, so this is not really a challenge. (It would be if you want to install Singular on an embedded system. But that's a challenge anyway.)
On the other hand the Singular team distributes fall-back binaries for those users which are not able to build Singular on their own. These binaries must not have any external dependencies, because system libraries vary from distro to distro.
What about the dynamic modules which already come with Singular?
Answer:
> ldd p_Procs_FieldQ.so
statically linked
This was a surprise. Since dynamic modules should make the binary small, I did not expect the modules itself to be linked statically, because - incorporating a whole bunch of system library stuff - now the modules become quite fat.
To be honest, until I typed that ldd command above, I thought, that shared and static are mutually exclusive. Indeed, the opposite of static is dynamic. Albeit the opposite of a shared library is a static one, a shared library itself could be linked dynamically or statically with the system libraries. So statically linked dynamic modules do make sense in the Singular context of providing fall-back binaries.
How can I archive this with pyobject.so? Just adding -shared -Xlinker -static to the Makefile did not do the job. The module was compiled and linked without error, but loading yielded an dlopen error: The symbol _Py_NoneStruct was not found.
A quick google search implied that there is something special with PyNone, so I avoided to use it. But then, the next unresolved symbol occurred. Ah, do'h! dlopen just complains about the first missing symbol. Actually, I never linked to libpython, because I put the libraries in the wrong order and (using -static) the order does matter. Also, linking like this does not complain about missing symbols, because the symbols might be resolved on runtime from the binary or other shared libraries. The latter was forbidden by the -static flag, and so, and so, and so, my module got corrupted.
Finally, I ended up with the following linkage call
g++ pyobject.dl_o -Xlinker -static -nodefaultlibs -Xlinker -export-dynamic -shared -L/usr/lib64/python2.6 -L/usr/lib64 -lpython2.6 -lpthread -ldl -lutil -lc -o pyobject.sowhere -lpthread -ldl -lutil fulfill the dependencies of libpython2.6 and the construct -nodefaultlibs ... -lc avoids trying to link against some unused shared c++-libs (which - of course - is not possible statically).
Note that this only works, if all libraries are available in their static lib***.a variants. Even more: to get position-independent code (i. e. shared-library conforming) each of which has to be compiled with -fPIC, which is not common for .a-libs. So ./configure --with-python tests for all of this and falls back to the classical dynamic-shared-modules in case of failure. Also, the user can force to build these light-weight modules by typing
./configure --with-python=module,dynamic
at the command prompt. In addition
./configure --with-python=embed
will incorporate the pyobject extension completely into the binary.
My best,
Alexander
P.S.: I'm aware, there are more dependencies besides libpython, for instance the Python standard library (written in Python itself), which needs to be bundled for providing a completely distro-independent binary distribution. But that's another story.
Thursday, December 2, 2010
Singular memory
As you probably know: Singular uses omalloc by Olaf Bachmann. It allows fast allocation of memory blocks of the same size.
Just a while ago we released new Singular: 3-1-2. Among other stuff it has the following experimental (and undocumented) feature: one can print out omalloc status (every single memory-related number) AND will try to test omalloc consistency.
In order to call it - simply type:
Here follows its output on my machine:
The later two lines indicate that no error was found.
PURPOSE: calling this function after a suspicious kernel function might indicate a memory corruption in it.
Just a while ago we released new Singular: 3-1-2. Among other stuff it has the following experimental (and undocumented) feature: one can print out omalloc status (every single memory-related number) AND will try to test omalloc consistency.
In order to call it - simply type:
system("omMemoryTest");Here follows its output on my machine:
> system("omMemoryTest");
[om_Info]:
MaxBytesSystem : 794624
CurrentBytesSystem : 794624
MaxBytesSbrk : 270336
CurrentBytesSbrk : 270336
MaxBytesMmap : 524288
CurrentBytesMmap : 524288
UsedBytes : 156976
AvailBytes : 493880
UsedBytesMalloc : 126568
AvailBytesMalloc : 0
MaxBytesFromMalloc : 126568
CurrentBytesFromMalloc : 126568
MaxBytesFromValloc : 524288
CurrentBytesFromValloc : 524288
UsedBytesFromValloc : 30408
AvailBytesFromValloc : 493880
MaxPages : 24
UsedPages : 23
AvailPages : 105
MaxRegionsAlloc : 1
CurrentRegionsAlloc : 1
[om_Opts]:
MinTrack : 0
MinCheck : 0
MaxTrack : 5
MaxCheck : 10
Keep : 100
HowToReportErrors : 2
MarkAsStatic : 0
PagesPerRegion : 128
OutOfMemoryFunc : 0x54e510
MemoryLowFunc : (nil)
ErrorHook : 0x877110
[om_ErrorStatus] : 'no error' (omError_NoError)
[om_InternalErrorStatus]: 'no error' (omError_NoError)
The later two lines indicate that no error was found.
PURPOSE: calling this function after a suspicious kernel function might indicate a memory corruption in it.
Wednesday, October 13, 2010
Welcome!
Welcome to Singular Team' newly created for blog for all of you interested in Singular and its development.
Let me explain: Singular is a computer algebra system for polynomial computations, with special emphasis on commutative and non-commutative algebra, algebraic geometry, and singularity theory. It is free and open-source under the GNU General Public Licence.
The aim with the blog is mostly to cover what is currently happening concerning the Singular development.
Also please let us know if there is anything you always wanted to know about Singular but were afraid to ask ;-)
Have fun!
Let me explain: Singular is a computer algebra system for polynomial computations, with special emphasis on commutative and non-commutative algebra, algebraic geometry, and singularity theory. It is free and open-source under the GNU General Public Licence.
The aim with the blog is mostly to cover what is currently happening concerning the Singular development.
Also please let us know if there is anything you always wanted to know about Singular but were afraid to ask ;-)
Have fun!
Subscribe to:
Posts (Atom)