asterisk / freepbx lock up and cannot find error

[mattmc97]

I actually did my proof of concept phone system on a plugpbx and now use it as a backup if my dual core atom mini-itx system dies, but....


I installed the following on a mini-itx system like I mentioned, and it has been running fine for a few months, but in the last few days it has begun to lockup.

Please see description below. I also posted this on the freepbx site, but I used the script that we have all developed here for 90% of the install.


Ubuntu 10.10
Asterisk 1.8.3.3
Freepbx 2.8.1.4

Phones: Linksys SPA-942s

We have a small office phone system and the last few days the system seems to lock up. The Freepbx webpage shows everything is okay, but the phone's lights are orange instead of green and there is no dial tone. All 5 phones on the network have the orange lights.

If I go into /var/log/asterisk/full there are no error messages, anywhere around the time it locked up and the only error I find is about the SPA3102 trying to register, but that happens all day log with no consequence.

If I do a "service freepbx restart", it will sit on "Stopping Asterisk" and will just sit there for a few minutes.

I then issue a Ctrl-C and "ps aux | grep asterisk, kill -9 the two asterisk processes; and it will then let me do a "service asterisk restart"

I have a 225 gig hard drive that is 10% full. No other errors logged in /var/log/asterisk/full.

Can someone please tell me where to look for errors?

TIA

mattmc

[PlugPBX Admin]

have asterisk run at maximum logging verbosity (instead of defaults)...

Or, run asterisk interactive from the console, with maximum debugging levels. Perhaps use 'screen' (apt-get install screen) to allow you to detach from the console to keep going.

Bonus points, use an application like 'mtee' or similar' to have the console output both sent to a log file and the console etc. You want to capture as much data around the silent crash as possible. Next steps are full out debugging tools etc etc.

[mattmc97]

Ok, so no problems for a month and now twice this week it happened again.

asterisk run at maximum logging verbosity (instead of defaults)

How do I do this?

I am using the freepbx reload script from here.

Code:

#!/bin/bash

AMPORTAL_BIN=/usr/local/sbin/amportal

if [ ! -x ${AMPORTAL_BIN} ]; then
        echo "error : amportal binary can not be found (${AMPORTAL_BIN})"
        exit 0
fi


start() {
        echo "Starting FreePBX ..."
        ${AMPORTAL_BIN} start
}

stop() {
        echo "Stopping FreePBX ..."
        ${AMPORTAL_BIN} stop
}

case "$1" in
  start)
        start
        ;;

  stop)
        stop
        ;;

  restart)
        stop
        start
        ;;

  *)
        echo $"Usage: $0 {start|stop|restart}"
        exit 1
esac

exit 0

Basically the lines are going orange on my SPA-941 phones.

The freepbx web interface will not respond and under 'Server Status' - Web Server is RED.

No errors in the /var/log/asterisk/full....

/var/log/asterisk/messages has not had an entry since May 4th.


Since it is only happening rarely, I don't think running asterisk interactive is the answer, as until this week, it has been almost a month since I had any issues.

What is strange is that 'service freepbx restart' cannot stop asterisk when the problem occurs. Many times, 'service freepbx restart' works perfectly. I just tried it now and asterisk seems to hang on 'STOPPING ASTERISK'.

Just got this error message:

<sup>STOPPING ASTERISK
*** glibc detected *** /usr/sbin/asterisk: double free or corruption (!prev): 0xb7377bd8 ***
======= Backtrace: =========
/lib/libc.so.6(+0x6c501)[0x239501]
/lib/libc.so.6(+0x6dd70)[0x23ad70]
/lib/libc.so.6(cfree+0x6d)[0x23de5d]
/usr/sbin/asterisk[0x808149c]
/usr/sbin/asterisk[0x80a677f]
/usr/sbin/asterisk[0x8079c3a]
/usr/sbin/asterisk[0x807a56f]
/usr/sbin/asterisk(ast_cli_command_full+0x134)[0x80c8ed4]
/usr/sbin/asterisk(ast_cli_command_multiple_full+0x7c)[0x80c914c]
/usr/sbin/asterisk[0x807ad43]
/usr/sbin/asterisk[0x819a87b]
/lib/libpthread.so.0(+0x5cc9)[0xf70cc9]
/lib/libc.so.6(clone+0x5e)[0x29d69e]
======= Memory map: ========
00110000-00134000 r-xp 00000000 08:01 10617027   /lib/libm-2.12.1.so
00134000-00135000 r--p 00023000 08:01 10617027   /lib/libm-2.12.1.so
00135000-00136000 rw-p 00024000 08:01 10617027   /lib/libm-2.12.1.so
00136000-0013c000 r-xp 00000000 08:01 10617030   /lib/libnss_compat-2.12.1.so
0013c000-0013d000 r--p 00006000 08:01 10617030   /lib/libnss_compat-2.12.1.so
0013d000-0013e000 rw-p 00007000 08:01 10617030   /lib/libnss_compat-2.12.1.so
0013e000-00148000 r-xp 00000000 08:01 10617032   /lib/libnss_files-2.12.1.so
00148000-00149000 r--p 00009000 08:01 10617032   /lib/libnss_files-2.12.1.so
00149000-0014a000 rw-p 0000a000 08:01 10617032   /lib/libnss_files-2.12.1.so
0014a000-0014d000 r-xp 00000000 08:01 10617199   /lib/libgpg-error.so.0.4.0
0014d000-0014e000 r--p 00002000 08:01 10617199   /lib/libgpg-error.so.0.4.0
0014e000-0014f000 rw-p 00003000 08:01 10617199   /lib/libgpg-error.so.0.4.0
0014f000-00154000 r-xp 00000000 08:01 400444     /usr/lib/asterisk/modules/res_monitor.so
00154000-00155000 r--p 00004000 08:01 400444     /usr/lib/asterisk/modules/res_monitor.so
00155000-00156000 rw-p 00005000 08:01 400444     /usr/lib/asterisk/modules/res_monitor.so
00156000-00166000 r-xp 00000000 08:01 400430     /usr/lib/asterisk/modules/res_agi.so
00166000-00167000 r--p 0000f000 08:01 400430     /usr/lib/asterisk/modules/res_agi.so
00167000-00169000 rw-p 00010000 08:01 400430     /usr/lib/asterisk/modules/res_agi.so
00169000-0017b000 r-xp 00000000 08:01 400440     /usr/lib/asterisk/modules/res_fax.so
0017b000-0017c000 r--p 00011000 08:01 400440     /usr/lib/asterisk/modules/res_fax.so
0017c000-0017d000 rw-p 00012000 08:01 400440     /usr/lib/asterisk/modules/res_fax.so
0017d000-00180000 r-xp 00000000 08:01 400454     /usr/lib/asterisk/modules/res_stun_monitor.so
00180000-00181000 r--p 00002000 08:01 400454     /usr/lib/asterisk/modules/res_stun_monitor.so
00181000-00182000 rw-p 00003000 08:01 400454     /usr/lib/asterisk/modules/res_stun_monitor.so
00184000-001c8000 r-xp 00000000 08:01 10616951   /lib/libssl.so.0.9.8
001c8000-001c9000 ---p 00044000 08:01 10616951   /lib/libssl.so.0.9.8
001c9000-001ca000 r--p 00044000 08:01 10616951   /lib/libssl.so.0.9.8
001ca000-001cd000 rw-p 00045000 08:01 10616951   /lib/libssl.so.0.9.8
001cd000-00324000 r-xp 00000000 08:01 10617022   /lib/libc-2.12.1.so
00324000-00326000 r--p 00157000 08:01 10617022   /lib/libc-2.12.1.so
00326000-00327000 rw-p 00159000 08:01 10617022   /lib/libc-2.12.1.so
00327000-0032a000 rw-p 00000000 00:00 0
0032a000-003c0000 r-xp 00000000 08:01 14029151   /usr/lib/libgnutls.so.26.14.12
003c0000-003c4000 r--p 00095000 08:01 14029151   /usr/lib/libgnutls.so.26.14.12
003c4000-003c5000 rw-p 00099000 08:01 14029151   /usr/lib/libgnutls.so.26.14.12
003c5000-003cc000 r-xp 00000000 08:01 10617041   /lib/librt-2.12.1.so
003cc000-003cd000 r--p 00006000 08:01 10617041   /lib/librt-2.12.1.so
003cd000-003ce000 rw-p 00007000 08:01 10617041   /lib/librt-2.12.1.so
003ce000-003d9000 r-xp 00000000 08:01 400452     /usr/lib/asterisk/modules/res_smdi.so
003d9000-003da000 r--p 0000a000 08:01 400452     /usr/lib/asterisk/modules/res_smdi.so
003da000-003db000 rw-p 0000b000 08:01 400452     /usr/lib/asterisk/modules/res_smdi.so
003db000-003e2000 r-xp 00000000 08:01 400428     /usr/lib/asterisk/modules/res_adsi.so
003e2000-003e3000 r--p 00006000 08:01 400428     /usr/lib/asterisk/modules/res_adsi.so
003e3000-003e4000 rw-p 00007000 08:01 400428     /usr/lib/asterisk/modules/res_adsi.so
003e4000-003e5000 r-xp 00000000 08:01 400390     /usr/lib/asterisk/modules/func_callcompletion.so
003e5000-003e6000 r--p 00000000 08:01 400390     /usr/lib/asterisk/modules/func_callcompletion.so
003e6000-003e7000 rw-p 00001000 08:01 400390     /usr/lib/asterisk/modules/func_callcompletion.so
003e7000-003ff000 r-xp 00000000 08:01 400442     /usr/lib/asterisk/modules/res_jabber.so
003ff000-00400000 r--p 00017000 08:01 400442     /usr/lib/asterisk/modules/res_jabber.so
00400000-00401000 rw-p 00018000 08:01 400442     /usr/lib/asterisk/modules/res_jabber.so
00401000-00402000 rw-p 00000000 00:00 0
00402000-00473000 r-xp 00000000 08:01 10617202   /lib/libgcrypt.so.11.5.3
00473000-00474000 r--p 00070000 08:01 10617202   /lib/libgcrypt.so.11.5.3
00474000-00476000 rw-p 00071000 08:01 10617202   /lib/libgcrypt.so.11.5.3
00476000-00479000 r-xp 00000000 08:01 400373     /usr/lib/asterisk/modules/format_wav.so
00479000-0047a000 r--p 00002000 08:01 400373     /usr/lib/asterisk/modules/format_wav.so
0047a000-0047b000 rw-p 00003000 08:01 400373     /usr/lib/asterisk/modules/format_wav.so
0047b000-0047c000 ---p 00000000 00:00 0
0047c000-004b7000 rwxp 00000000 00:00 0
004b8000-004cb000 r-xp 00000000 08:01 10617075   /lib/libz.so.1.2.3.4
004cb000-004cc000 r--p 00012000 08:01 10617075   /lib/libz.so.1.2.3.4
004cc000-004cd000 rw-p 00013000 08:01 10617075   /lib/libz.so.1.2.3.4
004cd000-004d7000 r-xp 00000000 08:01 400445     /usr/lib/asterisk/modules/res_musiconhold.so
004d7000-004d8000 r--p 00009000 08:01 400445     /usr/lib/asterisk/modules/res_musiconhold.so
004d8000-004d9000 rw-p 0000a000 08:01 400445     /usr/lib/asterisk/modules/res_musiconhold.so
004d9000-004db000 r-xp 00000000 08:01 400380     /usr/lib/asterisk/modules/cel_custom.so
004db000-004dc000 r--p 00001000 08:01 400380     /usr/lib/asterisk/modules/cel_custom.so
004dc000-004dd000 rw-p 00002000 08:01 400380     /usr/lib/asterisk/modules/cel_custom.so
004dd000-004e6000 r-xp 00000000 08:01 400354     /usr/lib/asterisk/modules/codec_lpc10.so
004e6000-004e7000 r--p 00008000 08:01 400354     /usr/lib/asterisk/modules/codec_lpc10.so
004e7000-004e8000 rw-p 00009000 08:01 400354     /usr/lib/asterisk/modules/codec_lpc10.soAsterisk ended with exit status 134
Asterisk exited on signal 6.

Automatically restarting Asterisk.

Asterisk Stopped

STOPPING FOP SERVER
Starting FreePBX ...


SETTING FILE PERMISSIONS
Permissions OK

STARTING ASTERISK
Asterisk ended with exit status 1
Asterisk died with code 1.
Automatically restarting Asterisk.
Asterisk Started
root@GulfTexPBX:/var/log/asterisk# Asterisk ended with exit status 1
Asterisk died with code 1.
Automatically restarting Asterisk.
Asterisk ended with exit status 1
Asterisk died with code 1.
Automatically restarting Asterisk.
Asterisk ended with exit status 1
Asterisk died with code 1.
Automatically restarting Asterisk.
Asterisk ended with exit status 1
Asterisk died with code 1.
Automatically restarting Asterisk.
Asterisk ended with exit status 1
Asterisk died with code 1.
Automatically restarting Asterisk.
Asterisk ended with exit status 1
Asterisk died with code 1.
Automatically restarting Asterisk.</sup>


Here is my 'ps aux | grep asterisk'....

_{root@mypbx:~# ps aux | grep asterisk
asterisk   891  0.1  0.6  46524 13208 ?        S    Aug23   7:14 /usr/sbin/apache2 -k start
asterisk  2039  0.0  0.5  44936 11204 ?        S    12:05   0:00 /usr/sbin/apache2 -k start
asterisk  2043  0.1  0.5  45204 11920 ?        S    12:06   0:02 /usr/sbin/apache2 -k start
asterisk  2055  0.0  0.5  45204 11668 ?        S    12:08   0:00 /usr/sbin/apache2 -k start
asterisk  2197  0.0  0.5  44936 11352 ?        S    12:14   0:00 /usr/sbin/apache2 -k start
asterisk  2230  0.1  0.5  44948 11796 ?        S    12:15   0:02 /usr/sbin/apache2 -k start
asterisk  2236  0.0  0.6  46816 13428 ?        S    12:15   0:00 /usr/sbin/apache2 -k start
asterisk  2243  0.0  0.5  44976 11340 ?        S    12:16   0:00 /usr/sbin/apache2 -k start
asterisk  2261  0.0  0.6  46848 13716 ?        S    12:16   0:01 /usr/sbin/apache2 -k start
root      2703  0.0  0.0   1896   468 pts/0    S    12:19   0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asterisk
root      4197  0.0  0.0   1896   304 pts/0    S    12:37   0:00 /bin/sh /usr/sbin/safe_asterisk -U asterisk -G asterisk
asterisk  4200  3.1  6.0 148024 124984 pts/0   Sl   12:37   0:02 /usr/sbin/asterisk -f -U asterisk -G asterisk -vvvg -c
root      4524  0.0  0.0   4004   768 pts/1    S+   12:38   0:00 grep --color=auto asterisk
asterisk  9717  0.1  0.5  44948 11828 ?        S    Aug25   2:20 /usr/sbin/apache2 -k start}


Once I kill -9 all the asterisk processes, I can then restart it and it works fine.


ANYONE HAVE AN IDEA WHAT MAY BE CAUSING THIS?

thanks.