Re: [squid-users] Squid3 and lots of FIN_WAIT1

On Mon, 16 Nov 2009 16:40:16 +0100, "David B." <haazeloud_at_gmail.com>
wrote:
> Hi Squid users,
>
> We're using squid as a reverse proxy cache. Server (debian lenny)
> running squid (from lenny stable / squid/3.0.STABLE8) seems to have some
> stange behaviour.
> For example, we've got lots of TIN_WAIT1 TCP Connexions and we can't
> figure why. :(
>
> # netstat -na | wc -l
> 20065
>
> # netstat -na | grep 'FIN_WAIT1' | wc -l
> 11255
>
> Around 50% of TCP sessions are in "FIN_WAIT1", other 40% are
"ETABLISHED".
>
> Theses connexions are from client of squid cache :
> Exemple, T.T.T.T is my public interface.
> x.y.z.w is a client IP v4 address.
>
> tcp 0 1 T.T.T.T:80 x.y.z.w:50530 FIN_WAIT1
>
> Did someone notice this before ?

I have not seen FIN_WAIT1 before, but often see FIN_WAIT like this when
Squid is receiving a lot of connections.

I think its due to squid using sockets for short times (non-persistent
connections) and moving on. The system TCP timeouts are much longer.

>
> We thought this will not be squid related, but kernel related. I can
> only see this on squid machines and not apache machines for example. :(
>
> We can also see errors in the kern.log like :
> kernel: [1675388.847059] Out of socket memory
>
> This seems to be linked. I assumed i've got too much TCP sessions for
> the kernel reserved memory.

I agree.

>
> Squid is behind an LVS load balancer, do you think my TCP problem is
> from there ? Because on the nex request, the same client could hit a
> different squid machine.

Maybe yes, maybe no.
I assume the LVS works on TCP-link connections so one persistent
connection is not broken up. Thats the only breakage that could make this
issue worse.

Normal LVS would be helping a bit by spreading the load off the problem
Squid boxes.

>
> Perhaps my load is too high and i need to tune kernel via sysctl, but i
> can't figure what to do. For now, i've tried several things and i can't
> solved this issue.

You may want to check:
 * persistent connections is turned on (squid.conf)
 * system socket timeouts
 * half-closed clients is off. (squid.conf)

Hopefully someone with real finger-time at high performance systems will
be able to point at specific tuning knobs which are helpful.

>
> Here's some squid stats :
> HTTP/1.0 200 OK
> Server: squid/3.0.STABLE8
> Mime-Version: 1.0
> Date: Mon, 16 Nov 2009 15:35:35 GMT
> Content-Type: text/plain
> Expires: Mon, 16 Nov 2009 15:35:35 GMT
> Last-Modified: Mon, 16 Nov 2009 15:35:35 GMT
> X-Cache: MISS from ww.xx.com
> X-Cache-Lookup: MISS from ww.xx.com:80
> Connection: close
>
> Squid Object Cache: Version 3.0.STABLE8
> Start Time: Mon, 09 Nov 2009 09:03:56 GMT
> Current Time: Mon, 16 Nov 2009 15:35:35 GMT
> Connection information for squid:
> Number of clients accessing cache: 254750
> Number of HTTP requests received: 337655902
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 32244.7

avg 537 req/sec. Nice :)

Are you able to provide CPU details sop we can add this to
http://wiki.squid-cache.org/KnowledgeBase/Benchmarks ?

Amos
Received on Mon Nov 16 2009 - 22:01:24 MST