Testing contributed libraries

A new Singular release is in the pipeline. In the new release, some libraries distributed as experimental previously will be updated and promoted to standard. We wanted to send an email to experimental library authors in order to remind them to

• send us the latest version of their libraries and
• check if the library conforms to the requirements for standard libraries.

Apart from the examples provided in the documentation, each library should provide a list of commands and the corresponding output to check the correctness of all code paths. These will be included in the Singular test suite to verify that the library works as intended on all platforms supported by Singular in future releases.

While working on the email, it became apparent that the instructions for adding tests for libraries are buried in the source code. For easy access before they are published in the online manual for the next release, here are the lines from the file Tst/README with instructions on providing test data.

To add a new test for a library/command to the test-suite:

• For most library files, two sets of test files need to be provided.
  • Short tests should test the essential functionality of the library/command in a relatively short time (say, in no more than 30s).
  • Long tests should test the functionality of the library/command in detail so that, if possible, all relevant cases/results are tested. Nevertheless, such a test should not run longer than, say, 10 minutes.
  If useful tests generally execute in a short time, it is enough to have short tests only.
• Each set should include
• a tst file containing the code to execute the test (details of the format of these files are explained below),
• and a res file with the corresponding output.
• The test files should have the following name convention (replace xx with library name):
• xx_s.tst: Singular code for short and basic tests
• xx_s.res: Output of xx_s.tst
• xx_l.tst: Singular code for long and extended tests 
• xx_l.res: Output of xx_l.res
or, alternatively:
• xx.tst: Singular code for short tests, only
• xx.res: Output of xx.tst

Rules for providing tst files:

1. tst files always start with the following three commands as preamble:
   

   LIB "tst.lib";
   tst_init();
   tst_ignore("CVS ID $Id$"); // or version number here

   tst_init() writes some general info to stdout (like date,  uname, hostname, version, etc.). The library tst.lib (contained in the Singular distribution) provides, among others, the routines tst_init() and tst_ignore().
   
2. tst files should end with the following statements:
   

   tst_status(1); $

   which enables (automatic) checks of the timing/memory performance of Singular.
   
3. All system-dependent output (like run-times, memory usages, pathnames, dates, etc.) should generally be avoided.
   
4. After time/memory critical sections of the tst files, the command
   

   tst_status();

   should be inserted. This enables (automatic) checks of the  timing/memory performance of Singular since the last call to tst_status() (resp. since the start-up of Singular).
   
5.  If system-dependent output can not be avodied, the routine tst_ignore() should be used:
   

   tst_ignore(val [, keyword]): 'val' can have arbitrary type for which a string conversion exists; if present, keyword must be one of the following strings: "time", "memory"

   tst_ignore() outputs 'val' by adding the following prefix:
   

   no keyword -- // tst_ignore:
   "time" keyword -- // tst_ignore: time:
   "memory" keyword -- // tst_ignore: memory:

   which causes automatic tests to ignore these lines when doing a diff on result files. 

An example of a short tst file can be found here:
http://www.singular.uni-kl.de:8002/trac/browser/trunk/Tst/Short/gmssin_s.tst

You can download the newest version of tst.lib from
ftp://www.mathematik.uni-kl.de/pub/Math/Singular/singular-cd/LIBRARIES/tst.lib