Thread process debugging method#
Overview#
When a Process on UNIX over-occupies CPU or Falls into a situation such as a Process Hang, the Process Debugging utility can be used, provided for each UNIX OS, to find the cause by analyzing the currently Running Call Stack.
UNIX Process Debugging methods are different for each utility and slightly different for each UNIX, so the Command is often confused. It explains how to view each thread stack, focusing on simple Commands.
Using pstack#
As a Utility provided by SUN and some other UNIX and LINUX, it is a very useful Command to view the Call Stack of the Running Process by Thread. When it is DB Hang, use it.
How to use#
-
LINUX
shell> pstack processid
-
Sun
shell> pstack -F processid
Example#
Example
$ uname -a
Linux as48-x64 2.6.9-89.ELsmp #1 SMP Mon Apr 20 10:33:05 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
$ whereis pstack
pstack: /usr/bin/pstack /usr/share/man/man1/pstack.1.gz
$ ps -ef | grep altibase | grep omegaman
altibase 16745 16454 0 13:55 pts/2 00:00:00 grep altibase
altibase 22978 1 0 Dec04 ? 00:00:07 /home/omegaman/altibase_home/bin/altibase -p boot from admin
$ pstack 22978 ## 22978 is a processid of altibase
Thread 41 (Thread 1084229984 (LWP 22979)):
#0 0x00000039070c2f56 in __select_nocancel () from /lib64/tls/libc.so.6
#1 0x0000000000f40f4b in idvClockThread::run ()
#2 0x0000000000ef797c in idtBaseThread::staticRunner ()
#3 0x0000003907706137 in start_thread () from /lib64/tls/libpthread.so.0
#4 0x00000039070c9f03 in clone () from /lib64/tls/libc.so.6
Thread 40 (Thread 1094719840 (LWP 22980)):
Using dbx#
Provided by AIX, HP, and SUN. You can check the Call Stack information of the Running Process through the dbx command, or the stack for each thread can be checked in the core file.
This section describes the method used on AIX as an example.
How to use#
-
Check the process id of the process in question.
shell> ps -eafl | grep altibase -
Attach to process.
shell> dbx -a -
View all thread information
(dbx) thread -
View individual thread information only
(dbx) thread current
(dbx) where -
Detach
(dbx) detachcaution
The connected process must be released by executing the detach command. Otherwise, if exited, it may end up to the target process, so be careful.
-
Exit
(dbx) quit
Example#
$ dbx -a 421920
Waiting to attach to process 421920 ...
Successfully attached to altibase.
warning: Directory containing altibase could not be determined.
Apply 'use' command to initialize source path.
Type 'help' for help.
reading symbolic information ...warning: no source compiled with -g
stopped in _event_sleep at 0x9000000004b6a84 ($t6)
0x9000000004b6a84 (_event_sleep+0xe8) e8410028 ld r2,0x28(r1)
(dbx) where
_event_sleep(??, ??, ??, ??, ??, ??) at 0x9000000004b6a84
_event_wait(??, ??) at 0x9000000004b6f5c
_cond_wait_local(??, ??, ??) at 0x9000000004c305c
_cond_wait(??, ??, ??) at 0x9000000004c3628
pthread_cond_timedwait(??, ??, ??) at 0x9000000004c3e38
run()() at 0x100187228
staticRunner(void*)() at 0x10007ea70
(dbx) list
no source file
(dbx) detach
Using gdb#
If gdb (GNU debugger) is installed, the stack information for each thread can be checked by using gdb.
How to use#
-
attach gdb to the target process
shell> $gdb $ALTIBASE_HOME/bin/altibase processid -
Information output for each thread
(gdb) info threads -
Output stack trace of all threads
(gdb) thread apply all bt -
Switch to a specific thread
(gdb) t 1 ## switch to thread 1 -
Output the current thread's stack
(gdb) bt ## stack output -
Quit
(gdb) quit
Example (altibase PID: 34569)#
$ gdb $ALTIBASE_HOME/bin/altibase 34567
(gdb) info threads
42 Thread 31 (LWP 4) 0x329614 in _poll ()
41 Thread 30 0x300cf8 in _lwp_sema_wait ()
40 Thread 29 0x300cf8 in _lwp_sema_wait ()
39 Thread 28 (LWP 29) 0x329614 in _poll ()
38 Thread 27 (LWP 28) 0x329614 in _poll ()
37 Thread 26 (LWP 27) 0x329614 in _poll ()
36 Thread 25 (LWP 26) 0x329614 in _poll ()
5 LWP 28 0x329614 in _poll ()
4 LWP 29 0x329614 in _poll ()
3 LWP 30 0x300cf8 in _lwp_sema_wait ()
* 2 Thread 1 (LWP 1) 0x329614 in _poll ()
1 LWP 1 0x329614 in _poll ()
(gdb) thread apply all bt ## Output stack trace of all threads
Thread 2 (Thread 1 (LWP 1)):
#0 0x329614 in _poll ()
#1 0x2f2548 in select_large_fdset ()
#2 0x286908 in __1cKidcManagerGselect6FpnGfd_set_pnOPDL_Time_Value__i_ ()
#3 0x978d8 in __1cMmmtThreadMgrIDispatch6M_nGIDE_RC__ ()
#4 0x945f4 in __1cGmmiMgrIMainLoop6F_nGIDE_RC__ ()
#5 0x929e0 in main ()
Thread 1 (LWP 1 ):
#0 0x329614 in _poll ()
#1 0x2f2548 in select_large_fdset ()
#2 0x286908 in __1cKidcManagerGselect6FpnGfd_set_pnOPDL_Time_Value__i_ ()
#3 0x978d8 in __1cMmmtThreadMgrIDispatch6M_nGIDE_RC__ ()
---Type to continue, or q to quit---
#4 0x945f4 in __1cGmmiMgrIMainLoop6F_nGIDE_RC__ ()
#5 0x929e0 in main ()
#0 0x329614 in _poll ()
(gdb) t 1 ## Switch to thread 1
(gdb) bt ## Output stack
(gdb) quit