Test Coverage Completion

What else needs to be tested

Return to project homepage.

Introduction

This paper comes as a synthesis for the 'NPTL Test Coverage' analysis.

The goal is to summarize, on a per-function basis, which NPTL features need more testing, in the following fields:

When a lack of testing is highlighted, samples will be written and submitted to the Open Source community.

This work is designed to complete the excellent work of the Open POSIX Test Suite project (later designed by PTS), which is included in the Linux Test Project (LTP). The new tests that are written will be released under the GNU General Public Licence and submitted for comments and addition to the PTS and / or the LTP (depending on whether they are specific to linux or not).

Glossary

Here are some definitions. You can also find the Open POSIX Test Suite definitions of those terms here.

Conformance test
This kind of software aims to verify that a function behaves as it is expected, according to a standard (i.e. POSIX*). The test will fail if the function does not meet the required behavior. This includes actions done by the function when the parameters are corrects, and the error codes returned when they are illegals.
Scalability test
This kind of software is designed to determine whether the function will behave correctly when it is used in a larger environment. This includes for example running on 32 or 64 CPU machines, with a huge amount of memory, with thousands of threads, etc... What has to be verified on one hand is that the function is still conformant, and that performances are good ( O(1) or linear, according to the operation) on the other hand.
Stress test
The goal of stress tests is to make low probability and complex problems appear by overloading the NPTL library. This can be done by launching all tests (conformance, scalability, ...) at the same time, many times, and with thousands of threads on a SMP or NUMA machine with many powerfull processors. It can also be done by writing new specific stress programs that interlace calls to the same routines or by using a software shaker that adds hazard to the execution of the test programs so that it creates random situations. The drawback is that reproducing the problem is not easy, but - at least - we know a problem exists. Using a NPTL trace mechanism may help analyzing and understanding the problem.
Performance test
This particular kind of test is a sort of benchmark. It repeats a certain operation a lot of times and measures the average execution time. This is only useful for comparison - between two different versions of a particular library, between two different libraries, between different hardwares (harder to analyse).

To Do

This section contains directives for futures releases of the document (including the site forum). Those directives are ideas, and maybe not everything will be included. You are encouraged to submit suggestions or comments (the e-mail address is at the end of this document).

List of functions

The NPTL contains about 146 public functions. In order to keep things readable, this document will be split into groups of functions (in the forum), categorized according to arbitrary criterias.

  1. Threads related functions
  2. Threads attributes related functions
  3. Scheduling control related functions
  4. Initialization and cancelation related functions
  5. Mutex related functions
  6. Read-write locks related functions
  7. Conditionnal variables related functions
  8. Spinlocks related functions
  9. Barriers related functions
  10. Semaphores related functions
  11. Thread Local Data (TLD) related functions
  12. Signals related functions
  13. Timers related functions
  14. Message queues related functions
  15. Other functions

Here is the alphabetical list of functions, and the categories which they belong to:

Function Category Status
__libc_current_sigrtmax Signals Analysis not started
__libc_current_sigrtmin Signals Analysis not started
flockfile Other Analysis not started
fork Scheduler control Planned: #3
ftrylockfile Other Analysis not started
funlockfile Other Analysis not started
getpid Other Planned: #1
mq_close Message queues Analysis not started
mq_getattr Message queues Analysis not started
mq_notify Message queues Analysis not started
mq_open Message queues Analysis not started
mq_receive Message queues Analysis not started
mq_send Message queues Analysis not started
mq_setattr Message queues Analysis not started
mq_timedreceive Message queues Analysis not started
mq_timedsend Message queues Analysis not started
mq_unlink Message queues Analysis not started
pthread_atfork Threads Planned: #6
pthread_attr_destroy Threads attributes Analysis not started
pthread_attr_getaffinity_np Threads attributes Analysis not started
pthread_attr_getdetachstate Threads attributes Analysis not started
pthread_attr_getguardsize Threads attributes Analysis not started
pthread_attr_getinheritsched Threads attributes Analysis not started
pthread_attr_getschedparam Threads attributes Analysis not started
pthread_attr_getschedpolicy Threads attributes Analysis not started
pthread_attr_getscope Threads attributes Analysis not started
pthread_attr_getstack Threads attributes Analysis not started
pthread_attr_getstackaddr Threads attributes Analysis not started
pthread_attr_getstacksize Threads attributes Analysis not started
pthread_attr_init Threads attributes Analysis not started
pthread_attr_setaffinity_np Threads attributes Analysis not started
pthread_attr_setdetachstate Threads attributes Analysis not started
pthread_attr_setguardsize Threads attributes Analysis not started
pthread_attr_setinheritsched Threads attributes Analysis not started
pthread_attr_setschedparam Threads attributes Analysis not started
pthread_attr_setschedpolicy Threads attributes Analysis not started
pthread_attr_setscope Threads attributes Analysis not started
pthread_attr_setstack Threads attributes Analysis not started
pthread_attr_setstackaddr Threads attributes Analysis not started
pthread_attr_setstacksize Threads attributes Analysis not started
pthread_barrier_destroy Barriers Analysis not started
pthread_barrier_init Barriers Analysis not started
pthread_barrier_wait Barriers Analysis not started
pthread_barrierattr_destroy Barriers Analysis not started
pthread_barrierattr_getpshared Barriers Analysis not started
pthread_barrierattr_init Barriers Analysis not started
pthread_barrierattr_setpshared Barriers Analysis not started
pthread_cancel Initialization and Cancellation Planned: #14
pthread_cleanup_pop Initialization and Cancellation Planned: #15
pthread_cleanup_pop_restore_np Initialization and Cancellation Analysis not started
pthread_cleanup_push Initialization and Cancellation Planned: #16
pthread_cleanup_push_defer_np Initialization and Cancellation Analysis not started
pthread_cond_broadcast Conditionnal variables
Status on2004-08-20
Analysis100%
Conformance100% complete
StressN/A
ScalabilityN/A
pthread_cond_destroy Conditionnal variables
Status on2004-08-16
Analysis100%
Conformance0% complete (2 remaining)
StressN/A
ScalabilityN/A
pthread_cond_init Conditionnal variables
Status on2004-06-10
Analysis100%
Conformance100% complete
Stress100% complete
Scalability100% complete
pthread_cond_signal Conditionnal variables
Status on2004-08-16
Analysis100%
Conformance100% complete
StressN/A
ScalabilityN/A
pthread_cond_timedwait Conditionnal variables
Status on2004-07-22
Analysis100%
Conformance100% complete
Stress100% complete
Scalability100% complete
pthread_cond_wait Conditionnal variables
Status on2004-08-13
Analysis100%
Conformance100% complete
Stress100% complete
Scalability100% complete
pthread_condattr_destroy Conditionnal variables Analysis not started
pthread_condattr_getclock Conditionnal variables Analysis not started
pthread_condattr_getpshared Conditionnal variables Analysis not started
pthread_condattr_init Conditionnal variables Analysis not started
pthread_condattr_setclock Conditionnal variables Analysis not started
pthread_condattr_setpshared Conditionnal variables Analysis not started
pthread_create Threads
Status on2004-08-19
Analysis100%
Conformance7 tests to write
StressN/A
Scalability1 test to write
pthread_detach Threads Planned: #15
pthread_equal Threads Planned: #14
pthread_exit Threads Planned: #13
pthread_getaffinity_np Scheduler control Analysis not started
pthread_getattr_np Threads attributes Analysis not started
pthread_getconcurrency Scheduler control Analysis not started
pthread_getcpuclockid Threads Analysis not started
pthread_getschedparam Scheduler control Analysis not started
pthread_getspecific Thread Local Data Planned: #28
pthread_join Threads Planned: #5
pthread_key_create Thread Local Data Planned: #26
pthread_key_delete Thread Local Data Planned: #29
pthread_kill Signals Planned: #8
pthread_mutex_destroy Mutex Planned: #12
pthread_mutex_init Mutex
Status on2004-05-12
Analysis100%
Conformance100% complete
Stress100% complete
Scalability100% complete
pthread_mutex_lock Mutex
Status on2004-05-25
Analysis100%
Conformance100% complete
Stress100% complete
Scalability100% complete
pthread_mutex_timedlock Mutex Analysis not started
pthread_mutex_trylock Mutex
Status on2004-08-20
Analysis100%
Conformance4 more tests needed
Stress1 more test needed
ScalabilityN/A
pthread_mutex_unlock Mutex
Status on2004-05-22
Analysis100%
Conformance100% complete
Stress100% complete
ScalabilityN/A
pthread_mutexattr_destroy Mutex Analysis not started
pthread_mutexattr_getpshared Mutex Analysis not started
pthread_mutexattr_gettype Mutex Analysis not started
pthread_mutexattr_init Mutex Analysis not started
pthread_mutexattr_setpshared Mutex Analysis not started
pthread_mutexattr_settype Mutex Analysis not started
pthread_once Initialization and Cancellation Planned: #4
pthread_rwlock_destroy Read-write locks Analysis not started
pthread_rwlock_init Read-write locks Analysis not started
pthread_rwlock_rdlock Read-write locks Analysis not started
pthread_rwlock_timedrdlock Read-write locks Analysis not started
pthread_rwlock_timedwrlock Read-write locks Analysis not started
pthread_rwlock_tryrdlock Read-write locks Analysis not started
pthread_rwlock_trywrlock Read-write locks Analysis not started
pthread_rwlock_unlock Read-write locks Analysis not started
pthread_rwlock_wrlock Read-write locks Analysis not started
pthread_rwlockattr_destroy Read-write locks Analysis not started
pthread_rwlockattr_getkind_np Read-write locks Analysis not started
pthread_rwlockattr_getpshared Read-write locks Analysis not started
pthread_rwlockattr_init Read-write locks Analysis not started
pthread_rwlockattr_setkind_np Read-write locks Analysis not started
pthread_rwlockattr_setpshared Read-write locks Analysis not started
pthread_self Threads Planned: #2
pthread_setaffinity_np Scheduler control Analysis not started
pthread_setcancelstate Initialization and Cancellation Analysis not started
pthread_setcanceltype Initialization and Cancellation Analysis not started
pthread_setconcurrency Scheduler control Analysis not started
pthread_setschedparam Scheduler control Analysis not started
pthread_setspecific Thread Local Data Planned: #27
pthread_sigmask Signals Planned: #9
pthread_spin_destroy Spinlocks Analysis not started
pthread_spin_init Spinlocks Analysis not started
pthread_spin_lock Spinlocks Analysis not started
pthread_spin_trylock Spinlocks Analysis not started
pthread_spin_unlock Spinlocks Analysis not started
pthread_testcancel Initialization and Cancellation Analysis not started
pthread_timedjoin_np Threads Analysis not started
pthread_tryjoin_np Threads Analysis not started
pthread_yield Scheduler control Planned: #7
raise Signals Planned: #11
sem_close Semaphores Planned: #24
sem_destroy Semaphores Planned: #23
sem_getvalue Semaphores Planned: #22
sem_init Semaphores Planned: #18
sem_open Semaphores Planned: #17
sem_post Semaphores Planned: #20
sem_timedwait Semaphores Analysis not started
sem_trywait Semaphores Planned: #21
sem_unlink Semaphores Planned: #25
sem_wait Semaphores Planned: #19
sigaction Signals Planned: #12
sigfillset Signals Analysis not started
sigprocmask Signals Planned: #10
sigtimedwait Signals Analysis not started
sigwait Signals Planned: #13
sigwaitinfo Signals Analysis not started
timer_create Timers Analysis not started
timer_delete Timers Analysis not started
timer_getoverr Timers Analysis not started
timer_gettime Timers Analysis not started
timer_settime Timers Analysis not started
vfork Scheduler control Analysis not started

* POSIX (R) is a registered trademark of the IEEE

Contact or questions: Sébastien Decugis