[squid-users] 2.4.20-aa and LARGE Squid process -> SIGSEGV

Hi!

Right now we're trying to implement a large scale Squid proxy on
Debian/testing. We're using 2.4.20-aa and Squid-2.4.7-1.

We're encountering sporadic crashes of the squid children (SIGSEGV,
signal 11). We were investigating in several directions:

* the Kernel has highmem support enabled (we have 2GB physical RAM and
  4 GB swap)

* we tried different kernels (with or without highmem support)

* we tried another squid version (Squid-2.4.6 from the "stable"
  distribution of Debian)

* We recompiled Squid as Debian Package using gcc-3.2, since --
  according to the FAQ -- squid may crash with signal 11 with
  optimization enabled when using gcc-2.95.4 (Debian uses gcc-2.95.4,
  but still build squid using -O!)

* we closely observed dmesg, messages and syslog. No oddities were
  found. Squid simply crashes with signal 11.

* we tried both ufs and aufs as cache filesystems, since the FAQ tells
  us the async I/O may have bugs. Yet, the crashes still occur.

* We use two identical machines -- the crash happens on both machines.
  Same CPU, disks, RAM, manufacturer, heck -- even the same series!
  Yet, the crashes still occur on both machines.
  
To no avail -- Squid simply crashes from time to time. It's
impossible to predict when.

Attached is the config we use (minus commentaries and empty lines --
note that we precautiously changed cache_mem to 500 MB BUT never
restarted; so we're still using cache_mem 700 MB).

If we raise our current setting of "cache_mem 700 MB" in any way (e.g.
to 1000 MB), the squid will eventually crash -- very often after about
1h. Note that we have 2GB of RAM.

Questions:

* What are we doing wrong?
* Are other people using Squid with 2GB of RAM? How?

# cat /proc/meminfo
        total: used: free: shared: buffers: cached:
Mem: 2119065600 2013810688 105254912 0 21864448 1035046912
Swap: 4293509120 2646016 4290863104
MemTotal: 2069400 kB
MemFree: 102788 kB
MemShared: 0 kB
Buffers: 21352 kB
Cached: 1008304 kB
SwapCached: 2484 kB
Active: 688220 kB
Inactive: 1003264 kB
HighTotal: 1703872 kB
HighFree: 91268 kB
LowTotal: 365528 kB
LowFree: 11520 kB
SwapTotal: 4192880 kB
SwapFree: 4190296 kB
BigFree: 0 kB

(Squid is "up" for about 1d now)

# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.80GHz
stepping : 7
cpu MHz : 2785.629
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 5557.45

# cat /proc/swaps
Filename Type Size Used Priority
/dev/sda6 partition 2096440 0 -1
/dev/sda7 partition 2096440 0 -2

Yes, I guess we should remove or at least reduce the swap, since a
swapping Squid process will suck incredibly.

$ cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0
proc /proc proc rw 0 0
devpts /dev/pts devpts rw 0 0
/dev/sdb5 /squid-cache0 xfs rw 0 0
/dev/sdc5 /squid-cache1 xfs rw 0 0
/dev/sdd5 /squid-data xfs rw 0 0
/dev/sda5 /boot ext3 rw 0 0

Filesystem Size Used Avail Use% Mounted on
/dev/sda8 13G 3.5G 8.6G 29% /
/dev/sdb5 17G 3.7G 14G 22% /squid-cache0
/dev/sdc5 17G 3.7G 14G 22% /squid-cache1
/dev/sdd5 17G 505M 17G 3% /squid-data
/dev/sda5 69M 9.0M 56M 14% /boot

$ cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: SEAGATE Model: ST318406LC Rev: 8A03
  Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 01 Lun: 00
  Vendor: SEAGATE Model: ST318406LC Rev: 8A03
  Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 02 Lun: 00
  Vendor: SEAGATE Model: ST318406LC Rev: 8A03
  Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 03 Lun: 00
  Vendor: SEAGATE Model: ST318406LC Rev: 8A03
  Type: Direct-Access ANSI SCSI revision: 03
Host: scsi0 Channel: 00 Id: 06 Lun: 00
  Vendor: PE/PV Model: 1x5 SCSI BP Rev: 0.34
  Type: Processor ANSI SCSI revision: 02
  
from the menuconfig output:

      (4GB) High Memory Support

This was based on the help in the kernel config:

"If the machine has between 1 and 4 Gigabytes physical RAM, then
answer "4GB" here."

      (3.5GB) User address space size
      [*] HIGHMEM I/O support

dmesg says:

Linux version 2.4.20aa1 (root@spidergirl) (gcc version 2.95.4 20011002
(Debian prerelease)) #1 Thu Dec 19 14:35:35 CET 2002
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
 BIOS-e820: 000000007fff0000 - 000000007fffec00 (ACPI data)
 BIOS-e820: 000000007fffec00 - 000000007ffff000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
1663MB HIGHMEM available.
384MB LOWMEM available.
...
Calibrating delay loop... 5557.45 BogoMIPS
Memory: 2069264k/2097088k available (1659k kernel code, 27440k
reserved, 651k data, 136k init, 1703872k highmem)
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode cache hash table entries: 131072 (order: 8, 1048576 bytes)
Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
Buffer-cache hash table entries: 131072 (order: 7, 524288 bytes)
Page-cache hash table entries: 524288 (order: 9, 2097152 bytes)
CPU: L1 I cache: 0K, L1 D cache: 8K
CPU: L2 cache: 512K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: After generic, caps: bfebfbff 00000000 00000000 00000000
CPU: Common caps: bfebfbff 00000000 00000000 00000000
CPU: Intel(R) Xeon(TM) CPU 2.80GHz stepping 07
...

$ top -u proxy

top - 10:55:57 up 1 day, 19:54, 3 users, load average: 0.00, 0.00, 0.00
Tasks: 88 total, 2 running, 86 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3% user, 15.6% system, 0.0% nice, 84.1% idle
Mem: 2069400k total, 1970396k used, 99004k free, 21768k buffers
Swap: 4192880k total, 2584k used, 4190296k free, 1011120k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ Command
18064 proxy 15 0 625m 625m 1344 R 0.3 30.9 25:33.18 squid
18065 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:00.00 squid
18066 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.09 squid
18067 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.05 squid
18068 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.11 squid
18069 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.87 squid
18070 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.87 squid
18071 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.96 squid
18072 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.95 squid
18073 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.73 squid
18074 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.04 squid
18075 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.73 squid
18076 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.17 squid
18077 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.14 squid
18078 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.17 squid
18079 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.85 squid
18080 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.24 squid
18081 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.78 squid
18082 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.98 squid
18083 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.17 squid
18084 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.89 squid
18085 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.93 squid
18086 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.89 squid
18087 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.67 squid
18088 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.92 squid
18089 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.07 squid
18090 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.78 squid
18091 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.85 squid
18092 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.12 squid
18093 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.04 squid
18094 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.03 squid
18095 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.81 squid
18096 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:03.13 squid
18097 proxy 15 0 625m 625m 1344 S 0.0 30.9 0:02.74 squid

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
Out of memory.
We wish to hold the whole sky,
But we never will.