[squid-users] Squid3 and lots of FIN_WAIT1

Hi Squid users,

We're using squid as a reverse proxy cache. Server (debian lenny)
running squid (from lenny stable / squid/3.0.STABLE8) seems to have some
stange behaviour.
For example, we've got lots of TIN_WAIT1 TCP Connexions and we can't
figure why. :(

# netstat -na | wc -l
20065

# netstat -na | grep 'FIN_WAIT1' | wc -l
11255

Around 50% of TCP sessions are in "FIN_WAIT1", other 40% are "ETABLISHED".

Theses connexions are from client of squid cache :
Exemple, T.T.T.T is my public interface.
x.y.z.w is a client IP v4 address.

tcp 0 1 T.T.T.T:80 x.y.z.w:50530 FIN_WAIT1

Did someone notice this before ?

We thought this will not be squid related, but kernel related. I can
only see this on squid machines and not apache machines for example. :(

We can also see errors in the kern.log like :
kernel: [1675388.847059] Out of socket memory

This seems to be linked. I assumed i've got too much TCP sessions for
the kernel reserved memory.

Squid is behind an LVS load balancer, do you think my TCP problem is
from there ? Because on the nex request, the same client could hit a
different squid machine.

Perhaps my load is too high and i need to tune kernel via sysctl, but i
can't figure what to do. For now, i've tried several things and i can't
solved this issue.

Here's some squid stats :
HTTP/1.0 200 OK
Server: squid/3.0.STABLE8
Mime-Version: 1.0
Date: Mon, 16 Nov 2009 15:35:35 GMT
Content-Type: text/plain
Expires: Mon, 16 Nov 2009 15:35:35 GMT
Last-Modified: Mon, 16 Nov 2009 15:35:35 GMT
X-Cache: MISS from ww.xx.com
X-Cache-Lookup: MISS from ww.xx.com:80
Connection: close

Squid Object Cache: Version 3.0.STABLE8
Start Time: Mon, 09 Nov 2009 09:03:56 GMT
Current Time: Mon, 16 Nov 2009 15:35:35 GMT
Connection information for squid:
        Number of clients accessing cache: 254750
        Number of HTTP requests received: 337655902
        Number of ICP messages received: 0
        Number of ICP messages sent: 0
        Number of queued ICP replies: 0
        Number of HTCP messages received: 0
        Number of HTCP messages sent: 0
        Request failure ratio: 0.00
        Average HTTP requests per minute since start: 32244.7
        Average ICP messages per minute since start: 0.0
        Select loop called: 1720466448 times, 0.365 ms avg
Cache information for squid:
        Hits as % of all requests: 5min: 97.6%, 60min: 97.9%
        Hits as % of bytes sent: 5min: 96.0%, 60min: 96.5%
        Memory hits as % of hit requests: 5min: 73.8%, 60min: 74.8%
        Disk hits as % of hit requests: 5min: 4.9%, 60min: 4.9%
        Storage Swap size: 60397908 KB
        Storage Swap capacity: 90.0% used, 10.0% free
        Storage Mem size: 4194284 KB
        Storage Mem capacity: 100.0% used, 0.0% free
        Mean Object Size: 67.08 KB
        Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
        HTTP Requests (All): 0.00000 0.00000
        Cache Misses: 0.00091 0.00091
        Cache Hits: 0.00000 0.00000
        Near Hits: 0.00091 0.00091
        Not-Modified Replies: 0.00000 0.00000
        DNS Lookups: 0.00000 0.00000
        ICP Queries: 0.00000 0.00000
Resource usage for squid:
        UP Time: 628299.345 seconds
        CPU Time: 68051.533 seconds
        CPU Usage: 10.83%
        CPU Usage, 5 minute avg: 9.56%
        CPU Usage, 60 minute avg: 8.77%
        Process Data Segment Size via sbrk(): 1027896 KB
        Maximum Resident Size: 0 KB
        Page faults with physical i/o: 2
Memory usage for squid via mallinfo():
        Total space in arena: 862404 KB
        Ordinary blocks: 794528 KB 53203 blks
        Small blocks: 0 KB 0 blks
        Holding blocks: 379248 KB 2048 blks
        Free Small blocks: 0 KB
        Free Ordinary blocks: 67875 KB
        Total in use: 1173776 KB 95%
        Total free: 67875 KB 5%
        Total size: 1241652 KB
Memory accounted for:
        Total accounted: 895030 KB 72%
        memPool accounted: 895030 KB 72%
        memPool unaccounted: 346621 KB 28%
        memPoolAlloc calls: 53284818867
        memPoolFree calls: 53271070253
File descriptor usage for squid:
        Maximum number of file descriptors: 65535
        Largest file desc currently in use: 52590
        Number of file desc currently in use: 50471
        Files queued for open: 1
        Available number of file descriptors: 15063
        Reserved number of file descriptors: 100
        Store Disk files open: 42769
Internal Data Structures:
        907161 StoreEntries
        243509 StoreEntries with MemObjects
        243363 Hot Object Cache Items
        900335 on-disk objects

Thank you.
David.
Received on Mon Nov 16 2009 - 15:40:02 MST