TRY THIS Corrupted double-linked list: 0x09098b80

[mcbsys]

* Running Asterisk Version : Asterisk 1.4.21.2
* Asterisk Source Version : 1.4.21.2
* Zaptel Source Version : 1.4.12.1
* Libpri Source Version : 1.4.8
* Addons Source Version : 1.4.7

Here's a new one (to me). I'm sending a fax to a PiaF box that uses a TDM400 card. In zapata.conf, I set "usecallerid=no" since the client doesn't subscribe to caller id. In the inbound route, I set Fax Detection Type = Zaptel, and left Pause After Answer blank. (I wanted to see if it could detect the fax while playing the IVR and/or ringing the operator.)

So I'm watching the call come in on the CLI and this is what I see:

Verbosity is at least 4
    -- Starting simple switch on 'Zap/3-1'
    -- Executing [s@from-zaptel:1] NoOp("Zap/3-1", "Entering from-zaptel with DID == ") in new stack
    -- Executing [s@from-zaptel:2] Ringing("Zap/3-1", "") in new stack
    -- Executing [s@from-zaptel:3] Set("Zap/3-1", "DID=s") in new stack
    -- Executing [s@from-zaptel:4] NoOp("Zap/3-1", "DID is now s") in new stack
    -- Executing [s@from-zaptel:5] GotoIf("Zap/3-1", "1?zapok:notzap") in new stack
    -- Goto (from-zaptel,s,8)
    -- Executing [s@from-zaptel:8] NoOp("Zap/3-1", "Is a Zaptel Channel") in new stack
    -- Executing [s@from-zaptel:9] Set("Zap/3-1", "CHAN=3-1") in new stack
    -- Executing [s@from-zaptel:10] Set("Zap/3-1", "CHAN=3") in new stack
    -- Executing [s@from-zaptel:11] Macro("Zap/3-1", "from-zaptel-3|s|1") in new stack
    -- Executing [s@from-zaptel:12] NoOp("Zap/3-1", "Returned from Macro from-zaptel-3") in new stack
    -- Executing [s@from-zaptel:13] Goto("Zap/3-1", "from-pstn|s|1") in new stack
    -- Goto (from-pstn,s,1)
    -- Executing [s@from-pstn:1] Set("Zap/3-1", "__FROM_DID=s") in new stack
    -- Executing [s@from-pstn:2] ExecIf("Zap/3-1", "1 |Set|CALLERID(name)=") in new stack
    -- Executing [s@from-pstn:3] Answer("Zap/3-1", "") in new stack
    -- Executing [s@from-pstn:4] Wait("Zap/3-1", "0") in new stack
    -- Executing [s@from-pstn:5] Set("Zap/3-1", "__CALLINGPRES_SV=allowed_not_screened") in new stack
    -- Executing [s@from-pstn:6] SetCallerPres("Zap/3-1", "allowed_not_screened") in new stack
    -- Executing [s@from-pstn:7] Goto("Zap/3-1", "timeconditions|1|1") in new stack
    -- Goto (timeconditions,1,1)
    -- Executing [1@timeconditions:1] GotoIfTime("Zap/3-1", "09:00-16:00|mon-thu|*|*?ext-group|300|1") in new stack
    -- Executing [1@timeconditions:2] Goto("Zap/3-1", "ivr-3|s|1") in new stack
    -- Goto (ivr-3,s,1)
    -- Executing [s@ivr-3:1] Set("Zap/3-1", "MSG=custom/MainMenu") in new stack
    -- Executing [s@ivr-3:2] Set("Zap/3-1", "LOOPCOUNT=0") in new stack
    -- Executing [s@ivr-3:3] Set("Zap/3-1", "__DIR-CONTEXT=default") in new stack
    -- Executing [s@ivr-3:4] Set("Zap/3-1", "_IVR_CONTEXT_ivr-3=") in new stack
    -- Executing [s@ivr-3:5] Set("Zap/3-1", "_IVR_CONTEXT=ivr-3") in new stack
    -- Executing [s@ivr-3:6] GotoIf("Zap/3-1", "1?begin") in new stack
    -- Goto (ivr-3,s,9)
    -- Executing [s@ivr-3:9] Set("Zap/3-1", "TIMEOUT(digit)=3") in new stack
    -- Digit timeout set to 3
    -- Executing [s@ivr-3:10] Set("Zap/3-1", "TIMEOUT(response)=5") in new stack
    -- Response timeout set to 5
    -- Executing [s@ivr-3:11] Set("Zap/3-1", "__IVR_RETVM=") in new stack
    -- Executing [s@ivr-3:12] ExecIf("Zap/3-1", "1|Background|custom/MainMenu") in new stack
    -- <Zap/3-1> Playing 'custom/MainMenu' (language 'en')
mypbx*CLI> *** glibc detected *** /usr/sbin/asterisk: corrupted double-linked list: 0x09098b80 ***
======= Backtrace: =========
/lib/libc.so.6[0x86191f]
/lib/libc.so.6(cfree+0x90)[0x865000]
/usr/sbin/asterisk[0x80b2504]
/usr/sbin/asterisk[0x80fc06b]
/lib/libpthread.so.0[0x9742ca]
/lib/libc.so.6(clone+0x5e)[0x8cae5e]
======= Memory map: ========
00101000-00126000 r-xp 00000000 08:02 1388906    /usr/lib/libk5crypto.so.3.1
00126000-00127000 rwxp 00025000 08:02 1388906    /usr/lib/libk5crypto.so.3.1
00127000-00128000 r-xp 00000000 08:02 1669619    /usr/lib/asterisk/modules/app_echo.so
00128000-00129000 rwxp 00000000 08:02 1669619    /usr/lib/asterisk/modules/app_echo.so
00129000-00156000 r-xp 00000000 08:02 1388908    /usr/lib/libgssapi_krb5.so.2.2
00156000-00157000 rwxp 0002d000 08:02 1388908    /usr/lib/libgssapi_krb5.so.2.2
00157000-00158000 r-xp 00000000 08:02 1669697    /usr/lib/asterisk/modules/func_base64.so
00158000-00159000 rwxp 00000000 08:02 1669697    /usr/lib/asterisk/modules/func_base64.so
00159000-001e9000 r-xp 00000000 08:02 1388907    /usr/lib/libkrb5.so.3.3
001e9000-001ec000 rwxp 0008f000 08:02 1388907    /usr/lib/libkrb5.so.3.3
001ec000-001fe000 r-xp 00000000 08:02 1382445    /usr/lib/libz.so.1.2.3
001fe000-001ff000 rwxp 00011000 08:02 1382445    /usr/lib/libz.so.1.2.3
001ff000-00202000 r-xp 00000000 08:02 1669583    /usr/lib/asterisk/modules/res_indications.so
00202000-00203000 rwxp 00003000 08:02 1669583    /usr/lib/asterisk/modules/res_indications.so
00203000-0020d000 r-xp 00000000 08:02 1669722    /usr/lib/asterisk/modules/format_mp3.so
0020d000-00210000 rwxp 00009000 08:02 1669722    /usr/lib/asterisk/modules/format_mp3.so
00210000-0022a000 rwxp 00210000 00:00 0
0022a000-00230000 r-xp 00000000 08:02 1669716    /usr/lib/asterisk/modules/func_strings.so
00230000-00231000 rwxp 00005000 08:02 1669716    /usr/lib/asterisk/modules/func_strings.so
00231000-00233000 r-xp 00000000 08:02 1669672    /usr/lib/asterisk/modules/app_zapras.so
00233000-00234000 rwxp 00001000 08:02 1669672    /usr/lib/asterisk/modules/app_zapras.so
00234000-00236000 r-xp 00000000 08:02 1669709    /usr/lib/asterisk/modules/func_logic.so
00236000-00237000 rwxp 00001000 08:02 1669709    /usr/lib/asterisk/modules/func_logic.so
00237000-00238000 r-xp 00000000 08:02 1669680    /usr/lib/asterisk/modules/codec_ulaw.so
00238000-00239000 rwxp 00001000 08:02 1669680    /usr/lib/asterisk/modules/codec_ulaw.so
00239000-00247000 r-xp 00000000 08:02 1669582    /usr/lib/asterisk/modules/res_features.so
00247000-00249000 rwxp 0000d000 08:02 1669582    /usr/lib/asterisk/modules/res_features.so
00249000-00284000 r-xp 00000000 08:02 3432125    /lib/libsepol.so.1
00284000-00285000 rwxp 0003a000 08:02 3432125    /lib/libsepol.so.1
00285000-0028f000 rwxp 00285000 00:00 0
0028f000-00291000 r-xp 00000000 08:02 1669601    /usr/lib/asterisk/modules/pbx_realtime.so
00291000-00292000 rwxp 00001000 08:02 1669601    /usr/lib/asterisk/modules/pbx_realtime.so
00292000-0029f000 r-xp 00000000 08:02 1669613    /usr/lib/asterisk/modules/app_dial.so
0029f000-002a0000 rwxp 0000c000 08:02 1669613    /usr/lib/asterisk/modules/app_dial.so
002a0000-002a2000 r-xp 00000000 08:02 1669698    /usr/lib/asterisk/modules/func_callerid.so
002a2000-002a3000 rwxp 00001000 08:02 1669698    /usr/lib/asterisk/modules/func_callerid.so
002a3000-002a5000 r-xp 00000000 08:02 1669647    /usr/lib/asterisk/modules/app_realtime.so
002a5000-002a6000 rwxp 00002000 08:02 1669647    /usr/lib/asterisk/modules/app_realtime.so
002a6000-002a8000 r-xp 00000000 08:02 1669620    /usr/lib/asterisk/modules/app_exec.so
002a8000-002a9000 rwxp 00002000 08:02 1669620    /usr/lib/asterisk/modules/app_exec.so
002a9000-002ac000 r-xp 00000000 08:02 1669648    /usr/lib/asterisk/modules/app_record.so
002ac000-002ad000 rwxp 00002000 08:02 1669648    /usr/lib/asterisk/modules/app_record.so
002ad000-002af000 r-xp 00000000 08:02 1669642    /usr/lib/asterisk/modules/app_privacy.so
002af000-002b0000 rwxp 00001000 08:02 1669642    /usr/lib/asterisk/modules/app_privacy.so
002b0000-002
Disconnected from Asterisk server
Executing last minute cleanups

root@mypbx:~ $ /usr/sbin/safe_asterisk: line 125:  5849 Aborted                 (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} >&/dev/${TTY} < /dev/${TTY}
Asterisk ended with exit status 134
Asterisk exited on signal 6.
cat: /var/run/asterisk.pid: No such file or directory
Automatically restarting Asterisk.
mpg123: no process killed

It's actually blown me out of the CLI a couple times without my choosing to exit; this is the first time it gave me a dump.

What does this mean and how do I make it stop? Do I need to recompile Asterisk, and if so, how do I do that?

Mark

 

[Capstone Works]

This is an OLD post, but we receivedthis error periodically yesterday, during normal call operations

*** glibc detected *** /usr/sbin/asterisk: corrupted double-linked list: 0x00007fb99c07a490 ***

/usr/sbin/safe_asterisk: line 159: 20312 Aborted (core dumped) nice -n $PRIORITY ${ASTSBINDIR}/asterisk -f ${CLIARGS} ${ASTARGS} > /dev/${TTY} 2>&1 < /dev/${TTY}

Asterisk ended with exit status 134

Automatically restarting Asterisk.
--------------------------------
error a few days ago ... and when it occurred ALL Active Calls Dropped; may be load or memory consumption related, although load is LOW <<5% and memory is <<75% consumed


Sanitized status-results.txt attached, as we;; as summary details, below.
--------------------------------
x Asterisk = ONLINE | Dahdi = ONLINE | MySQL = ONLINE x
x SSH = ONLINE | Apache = ONLINE | Iptables = ONLINE x
x Fail2ban = ONLINE | Internet = ONLINE | Ip6Tables = ONLINE x
x Disk Free = ADEQUATE| Mem Free = ADEQUATE| NTPD = ONLINE x
x SendMail = ONLINE | Samba = ONLINE | Webmin = ONLINE x
x Ethernet0 = N/A | Ethernet1 = N/A | Wlan0 = N/A x
x x
x PIAF Installed Version = 2.0.6.5 under *HARDWARE* x
x FreePBX Version = 2.11.0.38 x
x Running Asterisk Version = 11.8.1 x
x Asterisk Source Version = 11.8.1 x
x Dahdi Source Version = 2.9.0 x
x Libpri Source Version = 1.4.14 x
x IP Address = on eth0 x
x Operating System = CentOS release 6.5 (Final) x
x Kernel Version = 2.6.32-431.1.2.0.1.el6.x86_64 - 64 Bit x
x Incredible Version = 11.8 x
---------------------------------
Asterisk Information
x CDR-MYSQL = OK | CDR-CUSTOM = OK | Flite Engine = OK x
x G722 Codec = OK | G726 Codec = OK | Speex Codec = OK x
x Resample Codec = OK | Alaw Codec = OK | Ulaw Codec = OK x
x GSM Codec = OK | ILBC Codec = OK | Dahdi Codec = OK x
x Jabber Connect = N/A | Gtalk Channel = N/A | Silk Codec = OK x
x Siren7 Codec = OK | Siren14 Codec = OK | G729 Codec = N/A x
x SCCP-B Codec = N/A | Motif Codec = OK | x
x x
x Asterisk Uptime = 11 hours, 24 minutes, 37 seconds x
x System Uptime = 85 hours x
x IAX2 Registrations = 0 IAX2 registrations. x
x SIP Registrations = 0 SIP registrations. x
x Parked Calls = 1 parked call in total. x
x Installed ISO Version = 20650 x
x System Installed on = 2014-03-17T14:16-0500 x
x Kickstart Method = Normal ks kickstart x
x

 

[Capstone Works]

More details:

localhost*CLI> core restart gracefully
localhost*CLI> *** glibc detected *** /usr/sbin/asterisk: corrupted double-linked list: 0x0000000002aeaef0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3f47c75726]
/lib64/libc.so.6[0x3f47c7820a]
/usr/sbin/asterisk(io_context_destroy+0x11)[0x4c8a81]
/usr/lib/asterisk/modules/chan_sccp.so(+0x81419)[0x7f9ea7edd419]
/usr/sbin/asterisk(ast_module_shutdown+0xd2)[0x4cbb92]
/usr/sbin/asterisk[0x435855]
/usr/sbin/asterisk[0x436247]
/usr/sbin/asterisk(ast_cli_command_full+0x156)[0x47ec06]
/usr/sbin/asterisk(ast_cli_command_multiple_full+0x78)[0x47ee38]
/usr/sbin/asterisk[0x4366ba]
/usr/sbin/asterisk[0x53d2fb]
/lib64/libpthread.so.0[0x3f480077e1]
/lib64/libc.so.6(clone+0x6d)[0x3f47ce152d]
======= Memory map: ========
00400000-005cf000 r-xp 00000000 fd:00 1313059 /usr/sbin/asterisk
007ce000-007de000 rw-p 001ce000 fd:00 1313059 /usr/sbin/asterisk
007de000-00815000 rw-p 00000000 00:00 0
02535000-02c41000 rw-p 00000000 00:00 0 [heap]
3f47400000-3f4741e000 r-xp 00000000 fd:00 1975918 /lib64/ld-2.12.so
3f4761e000-3f4761f000 r--p 0001e000 fd:00 1975918 /lib64/ld-2.12.so

<SNIP>

3f53432000-3f53631000 ---p 00032000 fd:00 1975947 /lib64/libidn.so.11.6.1
3f53631000-3f53632000 rw-p 00031000 fd:00 1975947 /lib64/libidn.so.11.6.1
3f53c00000-3f53c27000 r-xp 00000000 fd:00 1975926 /lib64/libk5crypto.so.3.1
Disconnected from Asterisk server
Executing last minute cleanups

 

[billsimon]

Are you running SCCP endpoints? The chan_sccp.so module in the backtrace list stands out to me as something more likely than other things to cause a crash. So if you're not using that protocol, disable it with a noload line in the /etc/asterisk/modules.conf file.

 

[Capstone Works]

Interesting - I do not see chan_sccp.so module in the /etc/asterisk/modules.conf anywhere - where else could it come from?

 

[Capstone Works]

I DO see chan_skinny.so Skinny Client Control Protocol (Skinny) but chan_skinny.so is not in modules.conf either
-------------------------
pbx*CLI> module show like chan-sccp.so
Module Description Use Count
0 modules loaded
pbx*CLI>

pbx*CLI> module show like chan
Module Description Use Count
app_chanisavail.so Check channel availability 0
app_channelredirect.so Redirects a given channel to a dialplan 0
app_chanspy.so Listen to the audio of an active channel 0
app_dumpchan.so Dump Info About The Calling Channel 0
chan_agent.so Agent Proxy Channel 0
chan_bridge.so Bridge Interaction Channel 0
chan_dahdi.so DAHDI Telephony Driver w/PRI 0
chan_iax2.so Inter Asterisk eXchange (Ver 2) 0
chan_local.so Local Proxy Channel (Note: used internal 1
chan_mgcp.so Media Gateway Control Protocol (MGCP) 0
chan_mobile.so Bluetooth Mobile Device Channel Driver 0
chan_motif.so Motif Jingle Channel Driver 0
chan_multicast_rtp.so Multicast RTP Paging Channel 0
chan_nbs.so Network Broadcast Sound Support 0
chan_ooh323.so Objective Systems H323 Channel 0
chan_phone.so Linux Telephony API Support 0
chan_sip.so Session Initiation Protocol (SIP) 6
chan_skinny.so Skinny Client Control Protocol (Skinny) 0
chan_unistim.so UNISTIM Protocol (USTM) 0
func_channel.so Channel information dialplan functions 0
res_calendar_exchange.so Asterisk MS Exchange Calendar Integratio 0
21 modules loaded
fstapbx*CLI>

 

[billsimon]

It's chan_sccp.so with an underscore. It won't appear in modules.conf unless you put a line in there to exclude it. By default, modules.conf autoloads everything in the modules directory unless you put a noload line to tell it not to do so.

 

[Capstone Works]

This may well be the result of the SQLLite bug referenced here. http://pbxinaflash.com/community/index.php?threads/sqlite2-in-piaf.14864/#post-95573

I'll report back if we are indeed fixed.

Added the following to /etc/asterisk/modules.conf
; Don't load SQLite modules because of crashes with heavy call volumes
; SQLite is only used for the PIAF Registry and isn't needed for Asterisk
noload => cdr_sqlite.so
noload => res_config_sqlite.so
;