RE: [squid-users] squid crashes by itself and reboots automatically from Ragheb Rustom on 2009-11-29 (squid-users)

From: Ragheb Rustom <ragheb_at_smartelecom.org>
Date: Mon, 30 Nov 2009 07:54:15 +0200

Hi Mike,

These are the gdb bt of the core dumps that I got on other servers showing
the same behavior. It seems that are all failing at the strcmp. Any ideas
what could be causing this?

Server 1:

gdb /usr/sbin/squid core.2582
GNU gdb Fedora (6.8-24.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(no debugging symbols found)

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /usr/lib64/libtcmalloc.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libtcmalloc.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libssl.so.7...
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.7
Reading symbols from /lib64/libcrypto.so.7...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypto.so.7
Reading symbols from /lib64/libdl.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/librt.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libm.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libunwind.so.7...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libunwind.so.7
Reading symbols from /usr/lib64/libstdc++.so.6...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2

(no debugging symbols found)
Core was generated by `(squid) -D'.
Program terminated with signal 6, Aborted.
[New process 2582]
[New process 2613]
[New process 2601]
[New process 2612]
[New process 2615]
[New process 2606]
[New process 2607]
[New process 2605]
[New process 2610]
[New process 2604]
[New process 2614]
[New process 2603]
[New process 2608]
[New process 2602]
[New process 2609]
[New process 2599]
[New process 2611]
[New process 2598]
[New process 2600]
[New process 2597]
[New process 2596]
[New process 2593]
[New process 2595]
[New process 2592]
[New process 2594]
[New process 2590]
[New process 2591]
Missing separate debuginfos, use: debuginfo-install squid.x86_64
---Type <return> to continue, or q <return> to quit---
#0 0x0000003931832215 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003931832215 in raise () from /lib64/libc.so.6
#1 0x0000003931833d83 in abort () from /lib64/libc.so.6
#2 0x00000000004993ca in death ()
#3 <signal handler called>
#4 0x00000039318809c2 in strcmp () from /lib64/libc.so.6
#5 0x000000000043270e in ?? ()
#6 0x00000000004905b4 in ?? ()
#7 0x000000000049111b in ?? ()
#8 0x00000000004a69d1 in ?? ()
#9 0x00000000004a8fca in aioCheckCallbacks ()
#10 0x0000000000493137 in ?? ()
#11 0x0000000000438b0e in ?? ()
#12 0x0000000000469d6d in ?? ()
#13 0x000000393181e32a in __libc_start_main () from /lib64/libc.so.6
#14 0x0000000000409cb9 in drand48 ()
#15 0x00007fff93828708 in ?? ()
#16 0x000000000000001c in ?? ()
#17 0x0000000000000002 in ?? ()
#18 0x00007fff93828e92 in ?? ()
#19 0x00007fff93828e9a in ?? ()
#20 0x0000000000000000 in ?? ()

Server 2

gdb /usr/sbin/squid core.2591
GNU gdb Fedora (6.8-24.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(no debugging symbols found)

(no debugging symbols found)
Core was generated by `(squid) -D'.
Program terminated with signal 6, Aborted.
[New process 2591]
[New process 2609]
[New process 2624]
[New process 2616]
[New process 2622]
[New process 2621]
[New process 2618]
[New process 2620]
[New process 2614]
[New process 2623]
[New process 2615]
[New process 2607]
[New process 2613]
[New process 2611]
[New process 2610]
[New process 2606]
[New process 2605]
[New process 2601]
[New process 2619]
[New process 2617]
[New process 2612]
[New process 2608]
[New process 2604]
[New process 2603]
[New process 2602]
[New process 2600]
[New process 2599]
Missing separate debuginfos, use: debuginfo-install squid.x86_64
---Type <return> to continue, or q <return> to quit---
#0 0x0000003931832215 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x0000003931832215 in raise () from /lib64/libc.so.6
#1 0x0000003931833d83 in abort () from /lib64/libc.so.6
#2 0x00000000004993ca in death ()
#3 <signal handler called>
#4 0x00000039318809c2 in strcmp () from /lib64/libc.so.6
#5 0x000000000043270e in ?? ()
#6 0x00000000004905b4 in ?? ()
#7 0x000000000049111b in ?? ()
#8 0x00000000004a69d1 in ?? ()
#9 0x00000000004a8fca in aioCheckCallbacks ()
#10 0x0000000000493137 in ?? ()
#11 0x0000000000438b0e in ?? ()
#12 0x0000000000469d6d in ?? ()
#13 0x000000393181e32a in __libc_start_main () from /lib64/libc.so.6
#14 0x0000000000409cb9 in drand48 ()
#15 0x00007fffdec70c78 in ?? ()
#16 0x000000000000001c in ?? ()
#17 0x0000000000000002 in ?? ()
#18 0x00007fffdec70e92 in ?? ()
#19 0x00007fffdec70e9a in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb)

Ragheb Rustom
Smartelecom S.A.R.L
Sin el fil - Mar Elias Street
Absi Center - Ground Floor
Beirut, Lebanon
Telefax: +961-1-487275
Mobile: +961-3-286282
Email: ragheb_at_smartelecom.org

-----Original Message-----
From: Mike Marchywka [mailto:marchywka_at_hotmail.com]
Sent: Monday, November 30, 2009 1:50 AM
To: gkinkie_at_gmail.com; ragheb_at_smartelecom.org
Cc: squid-users_at_squid-cache.org
Subject: RE: [squid-users] squid crashes by itself and reboots automatically

----------------------------------------
> Date: Mon, 30 Nov 2009 00:24:38 +0100
> From: gkinkie_at_gmail.com
> To: ragheb_at_smartelecom.org
> CC: squid-users_at_squid-cache.org
> Subject: Re: [squid-users] squid crashes by itself and reboots
automatically
>
> On Sun, Nov 29, 2009 at 8:25 PM, Ragheb Rustom wrote:
>> Dear all,
>>
>> I have multiple servers running squid 2.7-stable7. Lately I have noticed
>> that most of these servers are crashing sometimes everyday creating a
core
>> dump file and then after creating the file restart by itself. After
reading
>> the core file with gdb and doing a backtrace I got the following info.
Can
>> anyone please help me identify what is going on with these squid servers.
>> You help is very valuable.
>
> Is there anything in cache.log? Possibly just the few lines before the
restart.

the default config file seems good about logging stuff.
Was that sparse stack trace typical or just the first
one you had? The only sig that was obvious was strcmp-
if it was blowing up there you either had pathological strings or memory
corruption or timing issue etc.
Not too informative until you at least know if it is
a consistent failure point.

>
>
> --
> /kinkie

_________________________________________________________________
Windows 7: I wanted simpler, now it's simpler. I'm a rock star.
http://www.microsoft.com/Windows/windows-7/default.aspx?h=myidea?ocid=PID247
27::T:WLMTAGL:ON:WL:en-US:WWL_WIN_myidea:112009
Received on Mon Nov 30 2009 - 05:54:51 MST

This archive was generated by hypermail 2.2.0 : Mon Nov 30 2009 - 12:00:04 MST