In Solaris, there are so many tools for analyzing and debugging every single part of the OS. the most common is the tools for Process Analysis. Since there are so many tools for process analysis, it can be helpful to group them into general. The general syntax is as follows:

$ ptool pid
$ ptool pid/lwpid

In summary:

  • prstat – For viewing overall process status<.li>
  • ps – To print process status and information
  • ptree – To print a process ancestry tree
  • pgrep; pkill – To match a process name; to send a signal
  • pstop; prun – To freeze a process; to continue a process
  • pwait – To wait for a process to finish
  • preap – To reap zombies
  • pstack – For inspecting stack backtraces
  • pmap – For viewing memory segment details
  • pfiles – For listing file descriptor details
  • ptime – For timing a command
  • psig – To list signal handlers
  • pldd – To list dynamic libraries
  • pflags; pcred – To list tracing flags; to list process credentials
  • pargs; pwdx – To list arguments, env; to list working directory
  • plockstat – For observing lock activity

The following will explain each tool in brief:

Process Tree: ptree

The process parent-child relationship can be displayed with the ptree command. By default, all processes within the same process group ID are displayed.

$ ptree 22961
301 /usr/lib/ssh/sshd
21571 /usr/lib/ssh/sshd
21578 /usr/lib/ssh/sshd
21580 -ksh
22961 /opt/filebench/bin/filebench
22962 shadow -a shadow -i 1 -s ffffffff10000000 -m /var/tmp/fbench9Ca
22963 shadow -a shadow -i 2 -s ffffffff10000000 -m /var/tmp/fbench9Ca
22964 shadow -a shadow -i 3 -s ffffffff10000000 -m /var/tmp/fbench9Ca
22965 shadow -a shadow -i 4 -s ffffffff10000000 -m /var/tmp/fbench9Ca

Grepping for Processes: pgrep

The pgrep command provides a convenient way to produce a process ID list matching certain criteria.

$ pgrep filebench
22968
22961
22966
22979
...

Killing Processes: pkill

The pkill command provides a convenient way to send signals to a list or processes matching certain criteria.

If the signal is not specified, the default is to send a SIGTERM.

Temporarily Stop a Process: pstop

A process can be temporarily suspended with the pstop command.

Making a Process Runnable: prun

A process can be made runnable with the prun command.

Wait for Process Completion: pwait

The pwait command blocks and waits for termination of a process.

Reap a Zombie Process: preap

A zombie process can be reaped with the preap command, which was added in Solaris 9.

Process Stack: pstack

The stacks of all or specific threads within a process can be displayed with the pstack command.

$ pstack 23154
23154: shadow -a shadow -i 193 -s ffffffff10000000 -m /var/tmp/fbench9Cai2S
----------------- lwp# 1 / thread# 1 --------------------
ffffffff7e7ce0f4 lwp_wait (2, ffffffff7fffe9cc)
ffffffff7e7c9528 _thrp_join (2, 0, 0, 1, 100000000, ffffffff7fffe9cc) + 38
0000000100018300 threadflow_init (ffffffff3722f1b0, ffffffff10000000, 10006a658, 0, 0,
1000888b0) + 184
00000001000172f8 procflow_exec (6a000, 10006a000, 0, 6a000, 5, ffffffff3722f1b0) + 15c
0000000100026558 main (a3400, ffffffff7ffff948, ffffffff7fffeff8, a4000, 0, 1) + 414
000000010001585c _start (0, 0, 0, 0, 0, 0) + 17c
----------------- lwp# 2 / thread# 2 --------------------
000000010001ae90 flowoplib_hog (30d40, ffffffff651f3650, 30d40, ffffffff373aa3b8, 1,
2e906) + 68
00000001000194a4 flowop_start (ffffffff373aa3b8, 0, 1, 0, 1, 1000888b0) + 408
ffffffff7e7ccea0 _lwp_start (0, 0, 0, 0, 0, 0)

The pstack command can be very useful for diagnosing process hangs or the status of core dumps. By default it shows a stack backtrace for all the threads within a process. It can also be used as a crude performance analysis technique; by taking a few samples of the process stack, you can often determine where the process is spending most of its time.

Process Memory Map: pmap -x

The pmap command inspects a process, displaying every mapping within the process address space. The amount of resident, nonshared anonymous, and locked memory is shown for each mapping. This allows you to estimate shared and private memory usage.

$ pmap -x 102908
102908: sh
Address Kbytes Resident Anon Locked Mode Mapped File
00010000 88 88 - - r-x-- sh
00036000 8 8 8 - rwx-- sh
00038000 16 16 16 - rwx-- [ heap ]
FF260000 16 16 - - r-x-- en_.so.2
FF272000 16 16 - - rwx-- en_US.so.2
FF280000 664 624 - - r-x-- libc.so.1
FF336000 32 32 8 - rwx-- libc.so.1
FF360000 16 16 - - r-x-- libc_psr.so.1
FF380000 24 24 - - r-x-- libgen.so.1
FF396000 8 8 - - rwx-- libgen.so.1
FF3A0000 8 8 - - r-x-- libdl.so.1
FF3B0000 8 8 8 - rwx-- [ anon ]
FF3C0000 152 152 - - r-x-- ld.so.1
FF3F6000 8 8 8 - rwx-- ld.so.1
FFBFE000 8 8 8 - rw--- [ stack ]
-------- ----- ----- ----- ------
total Kb 1072 1032 56 -

Process File Table: pfiles

A list of files open within a process can be obtained with the pfiles command.

# pfiles 21571
21571: /usr/lib/ssh/sshd
Current rlimit: 256 file descriptors
0: S_IFCHR mode:0666 dev:286,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_RDWR|O_LARGEFILE
/devices/pseudo/[email protected]:null
1: S_IFCHR mode:0666 dev:286,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_RDWR|O_LARGEFILE
/devices/pseudo/[email protected]:null
2: S_IFCHR mode:0666 dev:286,0 ino:6815752 uid:0 gid:3 rdev:13,2
O_RDWR|O_LARGEFILE
/devices/pseudo/[email protected]:null
3: S_IFCHR mode:0000 dev:286,0 ino:38639 uid:0 gid:0 rdev:215,2
O_RDWR FD_CLOEXEC
/devices/pseudo/[email protected]:crypto
4: S_IFIFO mode:0000 dev:294,0 ino:13099 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK FD_CLOEXEC
5: S_IFDOOR mode:0444 dev:295,0 ino:62 uid:0 gid:0 size:0
O_RDONLY|O_LARGEFILE FD_CLOEXEC door to nscd[89]
/var/run/name_service_door

Execution Time Statistics for a Process: ptime

A process can be timed with the ptime command for accurate microstate accounting instrumentation.

$ ptime sleep 1
real 1.203
user 0.022
sys 0.140

Process Signal Disposition: psig

A list of the signals and their current disposition can be displayed with psig.

$ psig $$
15481: -zsh
HUP caught 0
INT blocked,caught 0
QUIT blocked,ignored
ILL blocked,default
TRAP blocked,default
ABRT blocked,default
EMT blocked,default
FPE blocked,default
KILL default
BUS blocked,default
SEGV blocked,default
SYS blocked,default
PIPE blocked,default
ALRM blocked,caught 0
TERM blocked,ignored
USR1 blocked,default
USR2 blocked,default
CLD caught 0
PWR blocked,default
WINCH blocked,caught 0
URG blocked,default
POLL blocked,default
STOP default

Process Libraries: pldd

A list of the libraries currently mapped into a process can be displayed with pldd. This is useful for verifying which version or path of a library is being dynamically linked into a process.

$ pldd $$
482764: -ksh
/usr/lib/libsocket.so.1
/usr/lib/libnsl.so.1
/usr/lib/libc.so.1
/usr/lib/libdl.so.1
/usr/lib/libmp.so.2

Process Flags: pflags

The pflags command shows a variety of status information for a process. Information includes the mode 32-bit or 64-bit in which the process is running and the current state for each thread within the process. In addition, the top-level function on each threads stack is displayed.

$ pflags $$
482764: -ksh
data model = _ILP32 flags = PR_ORPHAN
/1: flags = PR_PCINVAL|PR_ASLEEP [ waitid(0x7,0x0,0xffbff938,0x7) ]

Process Credentials: pcred

The credentials for a process can be displayed with pcred.

$ pcred $$
482764: e/r/suid=36413 e/r/sgid=10
groups: 10 10512 570

Process Arguments: pargs

The full process arguments and optionally a list of the current environment settings can be displayed for a process with the pargs command.

$ pargs -ae 22961
22961: /opt/filebench/bin/filebench
argv[0]: /opt/filebench/bin/filebench
envp[0]: _=/opt/filebench/bin/filebench
envp[1]: MANPATH=/usr/man:/usr/dt/man:/usr/local/man:/opt/SUNWspro/man:/ws/on998-
tools/teamware/man:/home/rmc/local/man
envp[2]: VISUAL=/bin/vi
...

Process Working Directory: pwdx

The current working directory of a process can be displayed with the pwdx command.

$ pwdx 22961
22961: /tmp/filebench

Examining User-Level Locks in a Process

With the process lock statistics command, plockstat(1M) , you can observe hot lock behavior in user applications that use user-level locks. The plockstat command uses DTrace to instrument and measure lock statistics.

# plockstat -p 27088
^C
Mutex block
Count nsec Lock Caller
-------------------------------------------------------------------------------
102 39461866 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
4 21605652 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
11 19908101 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
12 16107603 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
10 9000198 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
14 5833887 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
10 5366750 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
120 964911 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
48 713877 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
52 575273 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
89 534127 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
14 427750 libaio.so.1`__aio_mutex libaio.so.1`_aio_lock+0x28
1 348476 libaio.so.1`__aio_mutex libaio.so.1`_aio_req_add+0x228
Mutex spin

Conclusion

using the above tools will help on figuring out how the process being handled by the OS and therefore, is a good start of troubleshooting the most common issues related to processes.