Re: [squid-users] segfault at 0

>> I'm having squid crash/restarting automatically from a week ago more
>> or less, without any hardware or software change.
>>
>> I had the squid 3.1.16 release which I updated yesterday to 3.1.19
>> from the gentoo package manager, and it keep logging the same errors:
>>
>> This is the evidence:
>>
>> firewall ~ # cat /var/log/kern.log | grep squid
>>
>> May 16 16:49:24 localhost kernel: squid[13702]: segfault at 0 ip
>> 000000000053eae3 sp 000003df3c6c39b8 error 4 in squid[400000+23e000]
>> May 16 16:49:24 localhost kernel: grsec: From 10.200.145.55:
>> Segmentation fault occurred at (null) in /usr/sbin/squid[squid:13702]
>> uid/euid:0/31 gid/egid:31/31, parent /usr/sbin squid[squid:12267]
>> uid/euid:0/0 gid/egid:31/31
>> May 16 16:49:28 localhost kernel: squid[14162]: segfault at 0 ip
>> 000000000053eae3 sp 000003a1d7901b98 error 4 in squid[400000+23e000]
>> May 16 16:49:28 localhost kernel: grsec: From 10.200.145.55:
>> Segmentation fault occurred at (null) in /usr/sbin/squid[squid:14162]
>> uid/euid:0/31 gid/egid:31/31, parent /usr/sbin squid[squid:12267]
>> uid/euid:0/0 gid/egid:31/31
>>
>> firewall ~ # cat /var/log/messages | grep squid
>>
>> May 16 16:49:24 localhost squid[12267]: Squid Parent: child process
>> 13702 exited due to signal 11 with status 0
>> May 16 16:49:27 localhost squid[12267]: Squid Parent: child process
>> 14162 started
>> May 16 16:49:28 localhost squid[12267]: Squid Parent: child process
>> 14162 exited due to signal 11 with status 0
>> May 16 16:49:31 localhost squid[12267]: Squid Parent: child process
>> 14201 started
>>
> <snip>
>
>> Any help will be appreciated.
>
>
> Can you find or make a core dump to see where this crash is occuring?
>
> There may also be some extra information in cache.log from Squid when it
> exited.
>
> http://wiki.squid-cache.org/SquidFaq/BugReporting
>
>
> Amos
>

Amos, thanks for your reply.

This is the debug information, I took it from the suggested method in
SquidFaq/BugReporting which was creating a script with the following code:

trap "rm -f $$.gdb" 0
cat <<EOF >$$.gdb
handle SIGPIPE pass nostop noprint
handle SIGTERM pass nostop noprint
handle SIGUSR1 pass nostop noprint
handle SIGSEGV stop
handle SIGABRT stop
run -DNYCd3
backtrace
quit
EOF
while sleep 2; do
  gdb -x $$.gdb /path/to/squid 2>&1 | tee -a squid.out
doneThis is the squid.out content after some crashes (it crashed a few
times, but I think it's the same information, so I'll just put once; if you
need it I'll post the entire file to an URL so you can check it).firewall ~
# cat /root/squid.out | less

GNU gdb (Gentoo 7.3.1 p2) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /usr/sbin/squid...(no debugging symbols found)...done.
2012/05/17 13:48:52| WARNING: -D command-line option is obsolete.
2012/05/17 13:48:52| Starting Squid Cache version 3.1.19 for
x86_64-pc-linux-gnu...
2012/05/17 13:48:52| Process ID 16781
2012/05/17 13:48:52| With 1024 file descriptors available
2012/05/17 13:48:52| Initializing IP Cache...
2012/05/17 13:48:52| DNS Socket created at [::], FD 10
2012/05/17 13:48:52| DNS Socket created at 0.0.0.0, FD 11
2012/05/17 13:48:52| Adding nameserver 127.0.0.1 from squid.conf
2012/05/17 13:48:52| Adding nameserver 208.67.222.222 from squid.conf
2012/05/17 13:48:52| Adding nameserver 208.67.220.220 from squid.conf
2012/05/17 13:48:52| helperOpenServers: Starting 20/20 'squidGuard'
processes
2012/05/17 13:48:52| Referer logging is disabled.
2012/05/17 13:48:52| Unlinkd pipe opened on FD 57
2012/05/17 13:48:52| Local cache digest enabled; rebuild/rewrite every
3600/3600 sec
2012/05/17 13:48:52| Store logging disabled
2012/05/17 13:48:52| Swap maxSize 16777216 + 524288 KB, estimated 1330884
objects
2012/05/17 13:48:52| Target number of buckets: 66544
2012/05/17 13:48:52| Using 131072 Store buckets
2012/05/17 13:48:52| Max Mem size: 524288 KB
2012/05/17 13:48:52| Max Swap size: 16777216 KB
2012/05/17 13:48:52| Version 1 of swap file with LFS support detected...
2012/05/17 13:48:52| Rebuilding storage in /var/cache/squid (CLEAN)
2012/05/17 13:48:52| Using Least Load store dir selection
2012/05/17 13:48:52| Current Directory is /root
2012/05/17 13:48:52| Loaded Icons.
2012/05/17 13:48:52| Accepting intercepted HTTP connections at
127.0.0.1:3128, FD 60.
2012/05/17 13:48:52| Accepting intercepted HTTP connections at
10.200.145.100:3128, FD 61.
2012/05/17 13:48:52| Accepting intercepted HTTP connections at
50.50.50.213:3128, FD 62.
2012/05/17 13:48:52| HTCP Disabled.
2012/05/17 13:48:52| Squid plugin modules loaded: 0
2012/05/17 13:48:52| Ready to serve requests.
2012/05/17 13:48:52| Store rebuilding is 8.92% complete
2012/05/17 13:48:52| Done reading /var/cache/squid swaplog (45906 entries)
2012/05/17 13:48:52| Finished rebuilding storage from disk.
2012/05/17 13:48:52| 45906 Entries scanned
2012/05/17 13:48:52| 0 Invalid entries.
2012/05/17 13:48:52| 0 With invalid flags.
2012/05/17 13:48:52| 45906 Objects loaded.
2012/05/17 13:48:52| 0 Objects expired.
2012/05/17 13:48:52| 0 Objects cancelled.
2012/05/17 13:48:52| 0 Duplicate URLs purged.
2012/05/17 13:48:52| 0 Swapfile clashes avoided.
2012/05/17 13:48:52| Took 0.30 seconds (151289.75 objects/sec).
2012/05/17 13:48:52| Beginning Validation Procedure
2012/05/17 13:48:52| Completed Validation Procedure
2012/05/17 13:48:52| Validated 91837 Entries
2012/05/17 13:48:52| store_swap_size = 1045568
2012/05/17 13:48:53| storeLateRelease: released 0 objects
2012/05/17 14:12:08| squidaio_queue_request: WARNING - Queue congestion
2012/05/17 14:28:40| WARNING: swapfile header inconsistent with available
data

Program received signal SIGSEGV, Segmentation fault.
0x000000000053eae3 in storeRead ()
#0 0x000000000053eae3 in storeRead ()
#1 0x000000000053fbe9 in store_client::fileRead() ()
#2 0x00000000005a1418 in UFSStoreState::readCompleted(char const*, int,
int, RefCount<ReadRequest>) ()
#3 0x00000000005b0d7c in DiskThreadsDiskFile::readDone(int, char const*,
int, int, RefCount<ReadRequest>) ()
#4 0x00000000005b0eca in DiskThreadsDiskFile::ReadDone(int, void*, char
const*, int, int) ()
#5 0x00000000005ada33 in DiskThreadsIOStrategy::callback() ()
#6 0x0000000000544615 in StoreHashIndex::callback() ()
#7 0x0000000000513166 in StoreRootEngine::checkEvents(int) ()
#8 0x00000000004c84d3 in EventLoop::checkEngine(AsyncEngine*, bool) ()
#9 0x00000000004c85fc in EventLoop::runOnce() ()
#10 0x00000000004c8758 in EventLoop::run() ()
#11 0x0000000000512a39 in SquidMain(int, char**) ()
#12 0x0000000000512fc3 in main ()
A debugging session is active.

        Inferior 1 [process 16781] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

This is the relevant (I think) information from squid/cache.log:

firewall ~ # cat /var/log/squid/cache.log | grep '2012/05/17 15:27'

2012/05/17 15:27:50| WARNING: swapfile header inconsistent with available
data
2012/05/17 15:27:52| Starting Squid Cache version 3.1.19 for
x86_64-pc-linux-gnu...
2012/05/17 15:27:52| Process ID 18570
2012/05/17 15:27:52| With 1024 file descriptors available
2012/05/17 15:27:52| Initializing IP Cache...
2012/05/17 15:27:52| DNS Socket created at [::], FD 10
2012/05/17 15:27:52| DNS Socket created at 0.0.0.0, FD 11
2012/05/17 15:27:52| Adding nameserver 127.0.0.1 from squid.conf
2012/05/17 15:27:52| Adding nameserver 208.67.222.222 from squid.conf
2012/05/17 15:27:52| Adding nameserver 208.67.220.220 from squid.conf
2012/05/17 15:27:52| helperOpenServers: Starting 20/20 'squidGuard'
processes
2012/05/17 15:27:52| Referer logging is disabled.
2012/05/17 15:27:52| Unlinkd pipe opened on FD 57
2012/05/17 15:27:52| Local cache digest enabled; rebuild/rewrite every
3600/3600 sec
2012/05/17 15:27:52| Store logging disabled
2012/05/17 15:27:52| Swap maxSize 16777216 + 524288 KB, estimated 1330884
objects
2012/05/17 15:27:52| Target number of buckets: 66544
2012/05/17 15:27:52| Using 131072 Store buckets
2012/05/17 15:27:52| Max Mem size: 524288 KB
2012/05/17 15:27:52| Max Swap size: 16777216 KB
2012/05/17 15:27:52| Version 1 of swap file with LFS support detected...
2012/05/17 15:27:52| Rebuilding storage in /var/cache/squid (DIRTY)
2012/05/17 15:27:52| Using Least Load store dir selection
2012/05/17 15:27:52| Current Directory is /root
2012/05/17 15:27:52| Loaded Icons.
2012/05/17 15:27:52| Accepting intercepted HTTP connections at
127.0.0.1:3128, FD 60.
2012/05/17 15:27:52| Accepting intercepted HTTP connections at
10.200.145.100:3128, FD 61.
2012/05/17 15:27:52| Accepting intercepted HTTP connections at
201.216.213.213:3128, FD 62.
2012/05/17 15:27:52| HTCP Disabled.
2012/05/17 15:27:52| Squid plugin modules loaded: 0
2012/05/17 15:27:52| Ready to serve requests.
2012/05/17 15:27:52| Store rebuilding is 8.56% complete
2012/05/17 15:27:53| Done reading /var/cache/squid swaplog (47855 entries)
2012/05/17 15:27:53| Finished rebuilding storage from disk.
2012/05/17 15:27:53| 47505 Entries scanned
2012/05/17 15:27:53| 0 Invalid entries.
2012/05/17 15:27:53| 0 With invalid flags.
2012/05/17 15:27:53| 47155 Objects loaded.
2012/05/17 15:27:53| 0 Objects expired.
2012/05/17 15:27:53| 350 Objects cancelled.
2012/05/17 15:27:53| 0 Duplicate URLs purged.
2012/05/17 15:27:53| 0 Swapfile clashes avoided.
2012/05/17 15:27:53| Took 0.31 seconds (150149.34 objects/sec).
2012/05/17 15:27:53| Beginning Validation Procedure
2012/05/17 15:27:53| Completed Validation Procedure
2012/05/17 15:27:53| Validated 94335 Entries
2012/05/17 15:27:53| store_swap_size = 1088120
2012/05/17 15:27:53| storeLateRelease: released 0 objects

This is the machine specs just in case it helps:

firewall ~ # uname -a
Linux firewall 2.6.34-hardened-r6 #1 SMP Tue Sep 28 11:09:50 ART 2010 x86_64
Intel(R) Xeon(R) CPU E5410 @ 2.33GHz GenuineIntel GNU/Linux

firewall ~ # cat /proc/meminfo
MemTotal: 2055888 kB
MemFree: 52544 kB
Buffers: 130540 kB
Cached: 1037776 kB

If the information is not correct, please let me know how to fix it.

Thank you!
Pablo
Received on Thu May 17 2012 - 18:45:59 MDT