Re: Large, busy, caches... problems?

Hi,

        Here at Connect we run 3 caches, each with 14Gb of disk, 768Mb of RAM,
2 CPU's, and about 15Gb of throughput a day per machine (including inter-cache
traffic sharing) and approx. 3000000 hits/day UDP/TCP per machine of which
40% is TCP (ie. ~15 TCP hits/second on average per machine). Our caches have
been rated as the fastest of the major ISP's in Australia by a significant
factor on average. It is with this background that I believe we can help you.

> The cache_swap setting has recently been reduced to 2000(MB) to see if
> this would improve performance. Previous settings of 4500(MB) and
> 16000(MB) have caused the cache to be unusable, with delays of >30
> seconds in retrieving local documents.
>
> With the 16GB cache (which never managed to grow beyond 8GB) it became
> obvious that there was insufficient memory to store the object meta
> data in memory. I had hoped that a reduction to 4.5GB might help.
> The reduction to 2GB has not helped completely either.

        How much memory do you have on your machine? It is my experience that
squid wants about 256 bytes of RAM per object in the cache, with the average
size of each object being about 16Kb on disk, so for 16GB of disk, you'd be
wanting at least 300Mb of RAM for indexing alone.

        Squid also wants about 32Mb of RAM to play in, and for in memory object
caching will want about 10MB per TCP hit/second (ie for us, 15 TCP hits/sec
= approx 150MB of RAM cache space). Our squid's eat up about 400MB of RAM in
full flight, and you will want almost as much again for the FTP processes, DNS
processes and general OS controlled disk biffer cache to provide good disk
throughput (hence our 768 MB RAM which is fully utilised).

        Reducing to 2GB of disk may still not help much if you're short of RAM,
because once you start getting the hits, squid will chew up RAM like nobody's
business trying to process them.

> The system seems to perform well until it starts cleaning. Once
> cleaning starts the system appears to begin paging, becoming worse
> over time (although this does not appear to be the only factor causing
> delays, the system is currently slow but does not appear to be paging
> [just my luck!]). Observations of the information reported by the
> cache manager interface suggest that the maximum resident size
> stabilises at around 80MB.

        We had the same problem. At 14Gb of disk, a highwater and lowwater
of 1% difference is still 140Mb, which is a lot to delete and can really tie
down the disks. Our simplest solution was to set the high and low water marks
to be the same value, that way squid cleans up as new data comes in which
averages out the disk load more evenly, so we found that we suffered an overall
5% disk throughput degeradation in doing this, as opposed to a 75% hit when
a major cleanup was initiated which caused the corresponding slowness in disk
response hence slowing down our caches much like you've seen.

> I would like to increase the cache size (2GB does not give us a
> particularly high hit rate compared to our other machines) - ideally
> to 4.5GB (or higher - the machine has 16GB available!). I have been
> told that if I can show that the problems will be 'resolved' by adding
> memory then it may be possible to purchase an additional 128MB RAM.

        Get more memory. I'd recommend 768MB ram MINIMUM if you're running
Irix, poss more (1GB recommended). We run Solaris 2.5.1 which in my experience
is a little less memory hungry than Irix (by about 10%).

> Would other users of large, busy, caches care to comment on their
> experiences? (Particularly with respect to the problems encountered
> when paging or operating during the clean cycle.)

        I've written a malloc library locally that concentrated memory access
patterns in a VM friendly way. This improved performance by a factor of 4
when memory started running tight. I understand the BSD 4.4 malloc library
does essentially whats required (GNU malloc sucks the puss seriously, it is
VERY bad for squid when memory's low, OK when there's plenty).

> In an attempt to reduce memory overheads, all FTP requests are farmed
> out to another of our machines.
>
> cache_mem = 4
> cache_swap = 2000

        Squid NEEDS RAM. You're attempting to cache LARGE volumes of data here,
which MUST be served quickly. There can be no substitute for more RAM. It
provides caching for the disks via the OS and indexing for all those objects.

> The disk cache is distributed over 8 XFS disk partitions (each
> partition on a separate physical disk)
>
> The machine in an SGI Challenge S, IP22, running IRIX 5.3 and is
> dedicated to running the cache.

        That sort of machine should cut it. We're running on UltraStation 2/170
with dual CPU's. Prob pretty close overall.

> I have tried 1.NOVM.8 but this crashes too frequently for use. It
> appears that the system runs out of file descriptors (2500 reported
> available upon startup).

> (My apologies for continued pleas for help - I want to get this thing
> working so that we can do some work with ICP with our other machines
> and child caches within the UK.)

        Our major saviour has been my implementation of async() systems calls
via threads which allows operations to be processed in the background readily.
Irix should support this too since it has kernel thread support. We have
achieved DRAMATIC speedups under heavy loads by threading the open(), close(),
unlink(), stat(), and other system calls, allowing us to serve files at over
200KB/sec speed while under full load (40 TCP hits/sec) and responding to
requests in under 1 second on average. I've sent the patches to Duane Wessels,
but I will see if can prepare a publicly available set for the current release.

        Cheers,

                Stew.

-- 
Stewart Forster (Security and Web Proxy Cache Administrator)
connect.com.au pty ltd, Level 9, 114 Albert Rd, Sth Melbourne, VIC 3205, Aust.
Email: slf@connect.com.au   Phone: +61 3 9251-3684   Fax: +61 3 9251-3666