NPTL Test Campaign #1

~ Report ~

NPTL Test Campaign - Introduction

* This document:

• NPTL Test Campaign - Introduction
  • * This document:
  • * Context:
  • * Goals:
  • * Methodology:
  • * Test machines:
• Conformance Tests - Results
• Scalability Tests - Results
• Scalability Tests - Graphics
  • * pthread_create.c:
  • * pthread_mutex_init.c:
• Stress Tests - Results
• Stress Tests - Graphics
• Conclusions

* Context:

* Goals:

1. Ensure that the tests are bug-free and can execute on different hardware and software.
2. Check the NPTL bugs, in several situations:
• bleeding-edge context, with vanilla kernel and glibc; or standard OS distributions
  
• with large machines (many CPUs, many I/O, lot of memory)
  
• with several CPU architectures (i686, Power-PC, Itanium-2)
3. Build a status on NPTL on main distributions, on the same machine (i686).
• Suse Linux Entreprise Server 9 (SLES9)
• RedHat Fedora Core 3 (test 2)
• RHEL3 (updated with NPTL and recent kernel)
  

* Methodology:

• testsuite-latest.patch will allow the tests to run against an alternative glibc (fresh compiled one, for example). It requires that the glibc was compiled with a specific script which can be found in the project forum. Some directory names may have to be changed, too.
• testsuite-SLES9PPC.patch is to be applied for the SLES9 test on PPC machine, as nm behaves differently there.
• testsuite-latest-ia64.patch is similar to the first patch, but for Itanium 2 machines (running BAS3 Linux).

* Test machines:

NPTL:

FASTBOOT:

VERONE:

STLINUX12:

Conformance Tests - Results

• Test Passed: The test could run and check that the assertion it tests is correctly supported in this implementation.
• Build Failure: The compilation failed. In many cases, this is due to a feature not supported in the implementation or at least missing in the header file (pthread.h for example).
• Link Failure: A problem occured at link time. This can be due to a feature declared in a header but no present in the library, for example.
• Link Skipped: Some of the test are "build-only" tests, and not meant to be run. They generally check a header file. Those tests do not contain a "main" function.
• Test Failed: The test could run and check that the assertion it tests is not correctly supported in this implementation -- or that the testcase is faulty.
• Test Untested: Some of the tests are able to detect that they are not suitable for this implementation. This is what they will return, meaning that they did not even try to test anything.
• Test Unresolved: Some error occured during the test execution, preventing the testcase to check its assertion.
• Test Unsupported: The feature this testcase tests is not provided by the implementation (but the compilation was OK).
• Test Killed: The return value from the testcase was not available to the test harness. This probably means that the test ended in a segmentation fault.
• Test Hung: Every test is run with a timeout of 120 seconds. When the testcase did not complete within this time period, it is killed with SIGALRM and reported as hung.
  

Scalability Tests - Results

NPTL 0.60

• PASS: The test executed and showed no scalability issue.
• OK: The test executed and a graph was generated.
  
• COMPIL: There was a compilation problem.
• FAIL: The test executed but a non-conformance or non-scalable issue was found.
  
• KILL: The test was killed, usually because of a segmentation fault.
• UNRSLV: The test was unresolved; this means that an unexpected error prevented the test to give a result PASS or FAIL.
  In the table only case, a pthread_barrier_init() operation failed.

Scalability Tests - Graphics

* pthread_create.c:

• No pthread_attr_* call:
  
• 0: Default
• Only one pthread_attr_* call:
  
• 1: Detached
• 2: Explicit Scheduling
• 3: FIFO Policy
• 4: RR policy
• 5: Max sched param
• 6: Min sched param
• 7: Alternative contension scope
• 8: Alternative stack
• 9: No guard size
• 10: 1 memory page guard size
• 11: Min stack size
• Stack combinations:
• 12: Min stack size, no guard
• 13: Min stack size, 1p guard
• 14: Detached, alternative stack
• 15: Detached, min stack size, no guard
• 16: Detached, min stack size, 1p guard
• Scheduling combinations:
• 17: Explicit FIFO, max param
• 18: Explicit RR, max param
• 19: Explicit FIFO, min param
• 20: Explicit RR, min param
• 21: Explicit FIFO, max param, alt scope
• 22: Explicit RR, max param, alt scope
• 23: Explicit FIFO, min param, alt scope
• 24: Explicit RR, min param, alt scope
• 25: Detached, explicit FIFO max param
• 26: Detached, explicit RR max param
• 27: Detached, explicit FIFO, min param
• 28: Detached, explicit RR, min param
• 29: Detached, explicit FIFO, max param, alt scope
• 30: Detached, explicit RR, max param, alt scope
• 31: Detached, explicit FIFO, min param, alt scope
• 32: Detached, explicit RR, min param, alt scope

2xi686 + latest glibc	Conformance issue - no graph was generated
2xi686 + SLES9
2xi686 + FC3T2	Conformance issue
8xi686 + FC3T2	Conformance issue
8xi686 + SLES9
4xPPC + SLES9
24xItanium2 + BAS3
24xItanium2 + latest glibc

* pthread_mutex_init.c:

2xi686 + latest glibc
2xi686 + SLES9
2xi686 + FC3T2
8xi686 + FC3T2
8xi686 + SLES9
4xPPC + SLES9
24xItanium2 + BAS3
24xItanium2 + latest glibc

Stress Tests - Results

2.6.7-BAS3V14

NPTL 0.60

• DEC: The free memory as reported by vmstat decreases regularly during the test execution.
• CONST: The free memory globally remains constant

• hand killed: This particular test did not terminate when it was killed with SIGUSR1, and had to be manually killed with SIGKILL.
• SEGV: The test terminated unexpectedly with a segmentation fault.

• the other status are the same as in the previous table.
  

Stress Tests - Graphics

Charge on the  CPUs.
Load average (over 1, 5 and 15 min)
Number of tasks

Conclusions

Ensure that the tests are bug-free and can execute on different hardware and software.

This first goal has been partly fulfilled. In each run, at least 5 testcases are killed with a segmentation fault. This may be due to bugs in the system or in the tests themselves. Further analysis is needed here.

Nevertheless, no unexpected problem was found in the 15 functions from our priority-1 list.

Check the NPTL bugs, in several situations:
 - bleeding-edge context, with vanilla kernel and glibc; or standard OS distributions
 - with large machines (many CPUs, many I/O, lot of memory)
 - with several CPU architectures (i686, Power-PC, Itanium 2)

The second goal was disturbed by the fact that a lot of POSIX functionnalities have been recently added to the Linux kernel and to glibc (message queues for example), and most of the tested distributions are older than these additions -- and therefore a lot of header and execution problems appeared with those routines.

It is also worth to notice that it was not possible to compile the library for Power-PC linux (as the binutils package failed to install), and that the process was far less straight-forward on Itanium-2 than on i686 (needed to add some links, etc...).

On the other hand, the NPTL behaved very well on large machines, showing no scalability weakness.

Build a status on NPTL on main distributions, on the same machine (i686).
 - Suse Linux Entreprise Server 9 (SLES9)
 - RedHat Fedora Core 3 (test 2)
 - RHEL3 (updated with NPTL support and recent kernel)

The conclusion for the third goal is that the more recent the distribution components (kernel, glibc), the better the results. Indeed, this is due to the NPTL being still under heavy development, with changes almost everyday; so a two-months delay between two Linux distributions leads to a large difference in results.

It is worth noting anyway that all the tested distributions have this header problem that prevented most of the scalability tests to be run; even if their NPTL binary file contains the functionnality. This is due to NPTL-only routines being unusable in default environment -- one need special linker flags to be able to use it (actually tests are linked to linuxthreads library at compilation time).

~ End ~

Author: Sébastien DECUGIS