Gdb

enable coredumps

echo '/var/crash/core/core.%e.%t.%p' > /proc/sys/kernel/core_pattern
echo 0 > /proc/sys/kernel/core_uses_pid
ulimit -c unlimited

core-files debuggen

# gdb /path/to/binary /path/to/corefile
[...]
gdb> bt
#0  0x00002abbafd9f265 in raise () from /lib64/libc.so.6
#1  0x00002abbafda0d10 in abort () from /lib64/libc.so.6
#2  0x00002abbafdd8beb in __libc_message () from /lib64/libc.so.6
#3 [...]
gdb> 

zombie prozesse finden

identifiziere zombie prozesse und die entsprechenden parent-PID's:

ps ax | awk '{print $1}' | grep -v "PID" \
  | xargs -n 1 ps lOp | grep -v "UID"    \
  | awk '{print"pid: "$3" *** parent_pid: "$4" *** status: "$10" *** process: "$13}' \
  | grep ": Z"

zombie prozesse los werden

how to kill zombies from process list
sadly the daemon/binary is poorly programmed and did NOT call 'wait' for its child.
so if daemon dies the child is a zombie and can not die!
use gdb or similar to do the wait-call!

Wenn HA ueber carp (freeBSD) konfiguriert ist, MUSS der server im state 'BACKUP' sein btw. darf nicht 'MASTER' sein siehe: Carp

to get a copy/paste-ready list of 'call wait()'-commands do:

ps aux | grep defunc | grep -v grep | awk '{ print "call wait("$2")" }'

now get PID of mother process and start gdb as follows:

# gdb /usr/local/sbin/ifstated [PID]
gdb> call wait([PID of zombie])

1. then 'CTRL-C'
2. then 'CTRL-D'
3. answer 'y'
4. program continues running ... childs dead... ;-)