Re: [squid-users] Store Disk files open

Marcelus Trojahn wrote:
> Hello,
>
> For a few weeks now I've been tuning a squid to my network. I have
> around 80Mb/s of HTTP traffic going through this server during most
> of the day. I had a little of every possible trouble I can think of
> and most of them are solved now after a few more white hair in my head
> :)
>
> Anyway... My only problem right now looks like some sort of disk I/O
> problem. Consider the following:
> File descriptor usage for squid:
> Maximum number of file descriptors: 32768
> Largest file desc currently in use: 24143
> Number of file desc currently in use: 23257
> Files queued for open: 2
> Available number of file descriptors: 9509
> Reserved number of file descriptors: 100
> Store Disk files open: 11399
>
> Squid isn't really slow or anything right now... But the value on the
> "Store disk files open" apparently never goes down and it is starting
> to concern me...
>
> Is there a way for some of this opened files to be stale, maybe?
> Should I start to worry about a incoming crash or I just need to
> recompile the whole thing with more max FDs?
>
> Here's a little more info just to show the current load:
>
> Squid Object Cache: Version 3.1.0.13-20090807
> Start Time: Fri, 21 Aug 2009 21:53:42 GMT
>
> Current Time: Sat, 22 Aug 2009 02:10:12 GMT
>
>
>
> Connection information for squid:
> Number of clients accessing cache: 1714
> Number of HTTP requests received: 6344801
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 24736.6
> Average ICP messages per minute since start: 0.0
> Select loop called: 189242493 times, 0.081 ms avg
> Cache information for squid:
> Hits as % of all requests: 5min: 31.9%, 60min: 36.2%
> Hits as % of bytes sent: 5min: 8.1%, 60min: 8.9%
> Memory hits as % of hit requests: 5min: 27.2%, 60min: 26.7%
> Disk hits as % of hit requests: 5min: 39.9%, 60min: 37.5%
> Storage Swap size: 97730572 KB
> Storage Swap capacity: 95.4% used, 4.6% free
> Storage Mem size: 4096128 KB
> Storage Mem capacity: 100.0% used, 1801439850948198.5% free
> Mean Object Size: 29.26 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.16775 0.23230
> Cache Misses: 0.23230 0.30459
> Cache Hits: 0.00091 0.02190
> Near Hits: 0.12106 0.25890
> Not-Modified Replies: 0.00000 0.00000
> DNS Lookups: 0.00190 0.01153
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 15389.656 seconds
> CPU Time: 6403.690 seconds
> CPU Usage: 41.61%
> CPU Usage, 5 minute avg: 36.83%
> CPU Usage, 60 minute avg: 41.76%
> Process Data Segment Size via sbrk(): 5909608 KB
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 8560
>
> The cache filesystem, after a bit of test, was chosen to be ext2 and
> squid was compiled like this:
>
> ./configure --prefix=/usr --build=x86_64-pc-linux-gnu
> --host=x86_64-pc-linux-gnu --mandir=/usr/share/man
> --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc
> --localstatedir=/var/lib --libdir=/usr/lib64 --sysconfdir=/etc/squid
> --libexecdir=/usr/libexec/squid --localstatedir=/var
> --datadir=/usr/share/squid --with-logdir=/var/log/squid
> --with-default-user=squid --enable-removal-policies=lru,heap
> --enable-external-acl-helpers=ip_user,session,unix_group
> --enable-useragent-log --disable-cache-digests --with-large-files
> --with-filedescriptors=32768 --disable-snmp --disable-ssl
> --disable-icap-client --enable-zph-qos --disable-ipv6 --enable-caps
> --enable-linux-netfilter --enable-async-io=36
>
> --
> Marcelus Trojahn

The easy way to tell is checking them all at:

   squidclient mgr:filedescriptors
   squidclient mgr:openfd_objects

Which give you a list of a) what every FD tracked by Squid is doing and
b) what the ones that apply to disk objects are doing.

[WARNING: detailed explanation of FD usage follows, you might not want
to know]

Your stats say this:
   Number of clients accessing cache: 1714
+
   Store Disk files open: 11399

Some of the files might be stale, yes.

However the FD accounted include TCP/UDP sockets etc.
  Each of the 1714 clients is using:
     1 for the link to the client
     1 for a disk/memory-backed object they are receiving/sending
     1 for the link to the server it came from (TCP_MISSes only)

So up to:
    1714 + 1714 + 1714 = 5142 FD will be in use for regular traffic.
    Add to that:
      the number of helper you have running.
      the number of *_port lines you have in squid.conf
      the number of DNS nameservers in use (for TCP DNS)
      the number of DNS requests made via UDP in the last 30 seconds or
so (your system UDP timeout tuning alters this) less those which have an
answer returned _multiplied_ by the number of nameservers in use.
      the number of files unlinkd is currently queued waiting for
erasure/RELEASE.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE6 or 3.0.STABLE18
   Current Beta Squid 3.1.0.13