Multithreaded Programming Guide

Chapter 5 Programming with the Operating Environment

This chapter describes how multithreading interacts with the Solaris operating environment and how the operating environment has changed to support multithreading.

Process Creation-Forking Issues

The default handling of the fork() function in the Solaris operating environment is somewhat different from the way fork() is handled in POSIX threads, although the Solaris operating environment does support both mechanisms.

Table 5-1 compares the differences and similarities of Solaris and pthreads fork() handling. When the comparable interface is not available either in POSIX threads or in Solaris threads, the `-' character appears in the table column.

Table 5-1 Comparing POSIX and Solaris fork() Handling


Solaris Operating Environment Interface 

POSIX Threads Interface 

Fork One Model 



Fork All Model 






The Fork One Model

As shown in Table 5-1, the behavior of the pthreads fork(2) function is the same as that of the Solaris fork1(2) function. Both the pthreads fork(2) function and the Solaris fork1(2) create a new process, duplicating the complete address space in the child, but duplicating only the calling thread in the child process.

This is useful when the child process immediately calls exec(), which is what happens after most calls to fork(). In this case, the child process does not need a duplicate of any thread other than the one that called fork().

In the child, do not call any library functions after calling fork() and before calling exec() because one of the library functions might use a lock that was held in the parent at the time of the fork(). The child process may execute only Async-Signal-Safe operations until one of the exec() handlers is called.

The Fork One Safety Problem and Solution

In addition to all of the usual concerns such as locking shared data, a library should be well-behaved with respect to forking a child process when only one thread is running (the one that called fork()). The problem is that the sole thread in the child process might try to grab a lock that is held by a thread that wasn't duplicated in the child.

This is not a problem most programs are likely to run into. Most programs call exec() in the child right after the return from fork(). However, if the program wishes to carry out some actions in the child before the call to exec(), or never calls exec(), then the child could encounter deadlock scenarios.

Each library writer should provide a safe solution, although not providing a fork-safe library is not a large concern because this condition is rare.

For example, assume that T1 is in the middle of printing something (and so is holding a lock for printf()), when T2 forks a new process. In the child process, if the sole thread (T2) calls printf(), it promptly deadlocks.

The POSIX fork() or Solaris fork1() duplicates only the thread that calls it. (Calling the Solaris fork() duplicates all threads, so this issue does not come up.)

To prevent deadlock, ensure that no such locks are being held at the time of forking. The most obvious way to do this is to have the forking thread acquire all the locks that could possibly be used by the child. Because you cannot do this for locks like those in printf() (because printf() is owned by libc), you must ensure that printf() is not being used at fork() time.

To manage the locks in your library:

In the following example, the list of locks used by the library is {L1,...Ln}, and the locking order for these locks is also L1...Ln.


In pthreads, you can add a call to pthread_atfork(f1, f2, f3) in your library's .init() section, where f1(), f2(), f3() are defined as follows:

f1() /* This is executed just before the process forks. */
 mutex_lock(L1); |
 mutex_lock(...); | -- ordered in lock order
 mutex_lock(Ln); |
 } V

f2() /* This is executed in the child after the process forks. */

f3() /* This is executed in the parent after the process forks. */

Another example of deadlock would be a thread in the parent process--other than the one that called Solaris fork1(2)--that has locked a mutex. This mutex is copied into the child process in its locked state, but no thread is copied over to unlock the mutex. So, any thread in the child that tries to lock the mutex waits forever.

Virtual Forks-vfork(2)

The standard vfork(2) function is unsafe in multithreaded programs. vfork(2) is like fork1(2) in that only the calling thread is copied in the child process. As in nonthreaded implementations, vfork() does not copy the address space for the child process.

Be careful that the thread in the child process does not change memory before it calls exec(2). Remember that vfork() gives the parent address space to the child. The parent gets its address space back after the child calls exec() or exits. It is important that the child not change the state of the parent.

For example, it is dangerous to create new threads between the call to vfork() and the call to exec(). This is an issue only if the fork one model is used, and only if the child does more than just call exec(). Most libraries are not fork-safe, so use pthread_atfork() to implement fork safety.

The Solution--pthread_atfork(3T)

Use pthread_atfork() to prevent deadlocks whenever you use the fork one model.

#include <pthread.h>

    void (*child) (void) );

The pthread_atfork() function declares fork() handlers that are called before and after fork() in the context of the thread that called fork().

Any one of these can be set to NULL. The order in which successive calls to pthread_atfork() are made is significant.

For example, a prepare handler could acquire all the mutexes needed, and then the parent and child handlers could release them. This ensures that all the relevant locks are held by the thread that calls the fork function before the process is forked, preventing the deadlock in the child.

Using the fork all model avoids the deadlock problem described in "The Fork One Safety Problem and Solution".

Return Values

pthread_atfork() returns a zero when it completes successfully. Any other returned value indicates that an error occurred. If the following condition is detected, pthread_atfork(3T) fails and returns the corresponding value.


Insufficient table space exists to record the fork handler addresses.

The Fork All Model

The Solaris fork(2) function duplicates the address space and all the threads (and LWPs) in the child. This is useful, for example, when the child process never calls exec(2) but does use its copy of the parent address space. The fork all functionality is not available in POSIX threads.

Note that when one thread in a process calls Solaris fork(2), threads that are blocked in an interruptible system call return EINTR.

Also, be careful not to create locks that are held by both the parent and child processes. This can happen when locks are allocated in memory that is sharable (that is mmap'ed with the MAP_SHARED flag). Note that this is not a problem if the fork one model is used.

Choosing the Right Fork

You determine whether fork() has a "fork all" or a "fork one" semantic in your application by linking with the appropriate library. Linking with -lthread gives you the "fork all" semantic for fork(), and linking with -lpthread gives the "fork one" semantic for fork() (see Figure 7-1for an explanation of compiling options).

Cautions for Any Fork

Be careful when using global state after a call to any fork() function.

For example, when one thread reads a file serially and another thread in the process successfully calls one of the forks, each process then contains a thread that is reading the file. Because the seek pointer for a file descriptor is shared after a fork(), the thread in the parent gets some data while the thread in the child gets the other. This introduces gaps in the sequential read accesses.

Process Creation-exec(2)and exit(2) Issues

Both the exec(2) and exit(2) system calls work as they do in single-threaded processes except that they destroy all the threads in the address space. Both calls block until all the execution resources (and so all active threads) are destroyed.

When exec() rebuilds the process, it creates a single lightweight process (LWP) . The process startup code builds the initial thread. As usual, if the initial thread returns, it calls exit() and the process is destroyed.

When all the threads in a process exit, the process exits. A call to any exec() function from a process with more than one thread terminates all threads, and loads and executes the new executable image. No destructor functions are called.

Timers, Alarms, and Profiling

The "End of Life" announcements for per-LWP timers (see timer_create(3R)) and per-thread alarms (see alarm(2) or setitimer(2)) were made in the Solaris 2.5 release. Both features are now supplemented with the per-process variants described in this section.

Originally, each LWP had a unique Realtime interval timer and alarm that a thread bound to the LWP could use. The timer or alarm delivered one signal to the thread when the timer or alarm expired.

Each LWP also had a virtual time or profile interval timer that a thread bound to the LWP could use. When the interval timer expired, either SIGVTALRM or SIGPROF, as appropriate, was sent to the LWP that owned the interval timer.

Per-LWP POSIX Timers

In the Solaris 2.3 and 2.4 releases, the timer_create(3R) function returned a timer object whose timer ID was meaningful only within the calling LWP and whose expiration signals were delivered to that LWP. Because of this, the only threads that could use the POSIX timer facility were bound threads.

Even with this restricted use, POSIX timers in Solaris 2.3 and 2.4 multithreaded applications were unreliable about masking the resulting signals and delivering the associated value from the sigvent structure.

With the Solaris 2.5 release, an application that is compiled defining the macro _POSIX_PER_PROCESS_TIMERS, or with a value greater that 199506L for the symbol _POSIX_C_SOURCE, can create per-process timers.

Applications compiled with a release before the Solaris 2.5 release, or without the feature test macros, will continue to create per-LWP POSIX timers. In some future release, calls to create per-LWP timers will return per-process timers.

The timer IDs of per-process timers are usable from any LWP, and the expiration signals are generated for the process rather than directed to a specific LWP.

The per-process timers are deleted only by timer_delete(3R) or when the process terminates.

Per-Thread Alarms

In the Solaris 2.3 and 2.4 releases, a call to alarm(2) or setitimer(2) was meaningful only within the calling LWP. Such timers were deleted automatically when the creating LWP terminated. Because of this, the only threads that could use these were bound threads.

Even with this restricted use, alarm() and setitimer() timers in Solaris 2.3 and 2.4 multithreaded applications were unreliable about masking the signals from the bound thread that issued these calls. When such masking was not required, then these two system calls worked reliably from bound threads.

With the Solaris 2.5 release, an application linking with -lpthread (POSIX) threads will get per-process delivery of SIGALRM when calling alarm(). The SIGALRM generated by alarm() is generated for the process rather than directed to a specific LWP. Also, the alarm is reset when the process terminates.

Applications compiled with a release before the Solaris 2.5 release, or not linked with -lpthread, will continue to see a per-LWP delivery of signals generated by alarm() and setitimer().

In some future release, calls to alarm() or to setitimer() with the ITIMER_REAL flag will cause the resulting SIGALRM to be sent to the process. For other flags, setitmer() will continue to be per-LWP. Flags other than the ITIMER_REAL flag, used by setitimer(), will continue to result in the generated signal being delivered to the LWP that issued the call, and so are usable only from bound threads.


You can profile each LWP with profil(2), giving each LWP its own buffer, or sharing buffers between LWPs. Profiling data is updated at each clock tick in LWP user time. The profile state is inherited from the creating LWP.

Nonlocal Goto--setjmp(3C) and longjmp(3C)

The scope of setjmp() and longjmp() is limited to one thread, which is fine most of the time. However, this does mean that a thread that handles a signal can longjmp() only when setjmp() is performed in the same thread.

Resource Limits

Resource limits are set on the entire process and are determined by adding the resource use of all threads in the process. When a soft resource limit is exceeded, the offending thread is sent the appropriate signal. The sum of the resources used in the process is available through getrusage(3B).

LWPs and Scheduling Classes

As mentioned in the "Scheduling" section of the "Covering Multithreading Basics", the Solaris pthreads implementation supports only the SCHED_OTHER scheduling policy. The others are optional under POSIX.

The POSIX SCHED_FIFO and SCHED_RR policies can be duplicated or emulated using the standard Solaris mechanisms. These scheduling mechanisms are described in this section.

The Solaris kernel has three classes of scheduling. The highest priority scheduling class is Realtime (RT). The middle priority scheduling class is system. The system class cannot be applied to a user process. The lowest priority scheduling class is timeshare (TS), which is also the default class.

Scheduling class is maintained for each LWP. When a process is created, the initial LWP inherits the scheduling class and priority of the creating LWP in the parent process. As more LWPs are created to run unbound threads, they also inherit this scheduling class and priority.

All unbound threads in a process have the same scheduling class and priority. Each scheduling class maps the priority of the LWP it is scheduling to an overall dispatching priority according to the configurable priority of the scheduling class.

Bound threads have the scheduling class and priority of their underlying LWPs. Each bound thread in a process can have a unique scheduling class and priority that is visible to the kernel. Bound threads are scheduled with respect to all other LWPs in the system.

Thread priorities regulate access to LWP resources. By default LWPs are in the timesharing class. For compute-bound multithreading, thread priorities are not very useful. For multithreaded applications that do a lot of synchronization using the MT libraries, thread priorities become more meaningful.

The scheduling class is set by priocntl(2). How you specify the first two arguments determines whether just the calling LWP or all the LWPs of one or more processes are affected. The third argument of priocntl() is the command, which can be one of the following.

Use priocntl() only on bound threads. To affect the priority of unbound threads, use pthread_setprio(3T).

Timeshare Scheduling

Timeshare scheduling distributes the processing resource fairly among the LWPs in this scheduling class. Other parts of the kernel can monopolize the processor for short intervals without degrading response time as seen by the user.

The priocntl(2) call sets the nice(2) level of one or more processes. The priocntl() call also affects the nice() level of all the timesharing class LWPs in the process. The nice() level ranges from 0 to +20 normally and from -20 to +20 for processes with superuser privilege. The lower the value, the higher the priority.

The dispatch priority of time-shared LWPs is calculated from the instantaneous CPU use rate of the LWP and from its nice() level. The nice() level indicates the relative priority of the LWPs to the timeshare scheduler.

LWPs with a greater nice() value get a smaller, but nonzero, share of the total processing. An LWP that has received a larger amount of processing is given lower priority than one that has received little or no processing.

Realtime Scheduling

The Realtime class (RT) can be applied to a whole process or to one or more LWPs in a process. This requires superuser privilege.

Unlike the nice(2) level of the timeshare class, LWPs that are classified Realtime can be assigned priorities either individually or jointly. A priocntl(2) call affects the attributes of all the Realtime LWPs in the process.

The scheduler always dispatches the highest-priority Realtime LWP. It preempts a lower-priority LWP when a higher-priority LWP becomes runnable. A preempted LWP is placed at the head of its level queue.

A Realtime LWP retains control of a processor until it is preempted, it suspends, or its Realtime priority is changed. LWPs in the RT class have absolute priority over processes in the TS class.

A new LWP inherits the scheduling class of the parent process or LWP. An RT class LWP inherits the parent's time slice, whether finite or infinite.

An LWP with a finite time slice runs until it terminates, blocks (for example, to wait for an I/O event), is preempted by a higher-priority runnable Realtime process, or the time slice expires.

An LWP with an infinite time slice ceases execution only when it terminates, blocks, or is preempted.

LWP Scheduling and Thread Binding

The threads library automatically adjusts the number of LWPs in the pool used to run unbound threads. Its objectives are:

Keep in mind that LWPs are time-sliced, not threads. This means that when there is only one LWP, there is no time slicing within the process--threads run on the LWP until they block (through interthread synchronization), are preempted, or terminate.

You can assign priorities to threads with pthread_setprio(3T); lower-priority unbound threads are assigned to LWPs only when no higher-priority unbound threads are available. Bound threads, of course, do not compete for LWPs because they have their own. Note that the thread priority that is set with pthread_setprio() regulates threads access to LWPs, not to CPUs.

Bind threads to your LWPs to get precise control over whatever is being scheduled. This control is not possible when many unbound threads compete for an LWP.

In particular, a lower-priority unbound thread could be on a higher priority LWP and running on a CPU, while a higher-priority unbound thread assigned to a lower-priority LWP is not running. In this sense, thread priorities are just a hint about access to CPUs.

Realtime threads are useful for getting a quick response to external stimuli. Consider a thread used for mouse tracking that must respond instantly to mouse clicks. By binding the thread to an LWP, you guarantee that there is an LWP available when it is needed. By assigning the LWP to the Realtime scheduling class, you ensure that the LWP is scheduled quickly in response to mouse clicks.

SIGWAITING--Creating LWPs for Waiting Threads

The library usually ensures that there are enough LWPs in its pool for a program to proceed.

When all the LWPs in the process are blocked in indefinite waits (such as blocked reading from a tty or network), the operating environment sends the new signal, SIGWAITING, to the process. This signal is handled by the threads library. When the process contains a thread that is waiting to run, a new LWP is created and the appropriate waiting thread is assigned to it for execution.

The SIGWAITING mechanism does not ensure that an additional LWP is created when one or more threads are compute bound and another thread becomes runnable. A compute-bound thread can prevent multiple runnable threads from being started because of a shortage of LWPs.

This can be prevented by calling thr_setconcurrency(3T). While using thr_setconcurrency() with POSIX threads is not POSIX compliant, its use is recommended to avoid LWP shortages for unbound threads in some computationally-intensive situations. (The only way to be completely POSIX compliant and avoid LWP shortages is to create only PTHREAD_SCOPE_SYSTEM bound threads.)

See "Thread Concurrency (Solaris Threads Only)"for more information about using the thr_setconcurrency(3T) function.

In Solaris threads, you can also use THR_NEW_LWP in calls to thr_create(3T) to create another LWP.

Aging LWPs

When the number of active threads is reduced, some of the LWPs in the pool are no longer needed. When there are more LWPs than active threads, the threads library destroys the unneeded LWPs. The library ages LWPs--they are deleted when they are unused for a "long" time, default is five minutes.

Extending Traditional Signals










Of particular importance to multithreaded programs is the effect of signals on pthread_cond_wait(3T). This call usually returns in response to a pthread_cond_signal(3T) or a pthread_cond_broadcast(3T), but, if the waiting thread receives a traditional UNIX signal, it returns with the error code EINTR. See "Interrupted Waits on Condition Variables (Solaris Threads Only)"for more information.

Synchronous Signals





Asynchronous Signals




Continuation Semantics



Example 5-1 Continuation Semantics

unsigned int nestcount;

unsigned int A(int i, int j) {

    if (i==0)
    else if (j==0)
        return(A(i-1, 1));

void sig(int i) {

main() {
    sigset(SIGINT, sig);

Operations on Signals













New sigwait() Implementations


Note -



#include <signal.h>

/* the Solaris 2.5 version*/
int sigwait(sigset_t *set);

/* the POSIX.1c version */






Note -

sigwait() should never be used with synchronous signals.



Thread-Directed Signals



A better way to deal with asynchronous signals is to treat them synchronously. By calling sigwait(2), discussed on "sigwait(2)", a thread can wait until a signal occurs.

Example 5-2 Asynchronous Signals and sigwait(2)

main() {
    sigset_t set;
    void runA(void);
    int sig;

    sigaddset(&set, SIGINT);
    pthread_sigsetmask(SIG_BLOCK, &set, NULL);

    while (1) {
        sigwait(&set, &sig);

void runA() {




Completion Semantics

Another way to deal with signals is with completion semantics.



Example 5-3 Completion Semantics

sigjmp_buf jbuf;
void mult_divide(void) {
    int a, b, c, d;
    void problem();

    sigset(SIGFPE, problem);
    while (1) {
        if (sigsetjmp(&jbuf) == 0) {
            d = a*b/c;

void problem(int sig) {
    siglongjmp(&jbuf, 1);







Completion semantics are often used to deal with exceptions. In particular, the Ada\256 programming language uses this model.

Note -


Signal Handlers and Async-Signal Safety










Table 5-2 Async-Signal-Safe Functions









































































Interrupted Waits on Condition Variables (Solaris Threads Only)




This implies that the mutex is locked in the signal handler because the handler might have to clean up after the thread. While this is true in the Solaris 2.5 release, it might change in the future, so do not rely upon this behavior.

Note -


Handler cleanup is illustrated by Example 5-4.

Example 5-4 Condition Variables and Interrupted Waits

int sig_catcher() {
    sigset_t set;
    void hdlr();


    sigaddset(&set, SIGINT);
    sigsetmask(SIG_UNBLOCK, &set, 0);

        /* signal occurred and lock is held */

void hdlr() {
    /* lock is held in the handler */



I/O Issues

One of the attractions of multithreaded programming is I/O performance. The traditional UNIX API gave you little assistance in this area--you either used the facilities of the file system or bypassed the file system entirely.

This section shows how to use threads to get more flexibility through I/O concurrency and multibuffering. This section also discusses the differences and similarities between the approaches of synchronous I/O (with threads) and asynchronous I/O (with and without threads).

I/O as a Remote Procedure Call

In the traditional UNIX model, I/O appears to be synchronous, as if you were placing a remote procedure call to the I/O device. Once the call returns, then the I/O has completed (or at least it appears to have completed--a write request, for example, might merely result in the transfer of the data to a buffer in the operating environment).

The advantage of this model is that it is easy to understand because, as a programmer you are very familiar with the concept of procedure calls.

An alternative approach not found in traditional UNIX systems is the asynchronous model, in which an I/O request merely starts an operation. The program must somehow discover when the operation completes.

This approach is not as simple as the synchronous model, but it has the advantage of allowing concurrent I/O and processing in traditional, single-threaded UNIX processes.

Tamed Asynchrony


Asynchronous I/O

In most situations there is no need for asynchronous I/O, since its effects can be achieved with the use of threads, with each thread doing synchronous I/O. However, in a few situations, threads cannot achieve what asynchronous I/O can.

The most straightforward example is writing to a tape drive to make the tape drive stream. Streaming prevents the tape drive from stopping while it is being written to and moves the tape forward at high speed while supplying a constant stream of data that is written to tape.

To do this, the tape driver in the kernel must issue a queued write request when the tape driver responds to an interrupt that indicates that the previous tape-write operation has completed.

Threads cannot guarantee that asynchronous writes will be ordered because the order in which threads execute is indeterminate. Specifying the order of a write to a tape, for example, is not possible.

Asynchronous I/O Operations

#include <sys/asynch.h>

    int whence, aio_result_t *resultp);

    off_t offset, int whence, aio_result_t *resultp);

aio_result_t *aiowait(const struct timeval *timeout);
int aiocancel(aio_result_t *resultp);



int aio_return;
int aio_errno;







Shared I/O and New I/O System Calls



#include <sys/types.h>
#include <unistd.h>


    off_t offset);


Alternatives to getc(3S) and putc(3S)

An additional problem occurs with standard I/O. Programmers are accustomed to routines such as getc(3S) and putc(3S) being very quick--they are implemented as macros. Because of this, they can be used within the inner loop of a program with no concerns about efficiency.

However, when they are made thread safe they suddenly become more expensive--they now require (at least) two internal subroutine calls, to lock and unlock a mutex.

To get around this problem, alternative versions of these routines are supplied, getc_unlocked(3S) and putc_unlocked(3S).

These do not acquire locks on a mutex and so are as quick as the original, nonthread-safe versions of getc(3S) and putc(3S).

However, to use them in a thread-safe way, you must explicitly lock and release the mutexes that protect the standard I/O streams, using flockfile(3S) and funlockfile(3S). The calls to these latter routines are placed outside the loop, and the calls to getc_unlocked() or putc_unlocked() are placed inside the loop.