Re: [squid-users] "Quadruple" memory usage with squid

On Mon, 23 Nov 2009 09:22:42 -0500, Linda Messerschmidt
<linda.messerschmidt_at_gmail.com> wrote:
> On Sun, Nov 22, 2009 at 9:03 PM, Henrik Nordstrom
> <henrik_at_henriknordstrom.net> wrote:
>> What is your average object size?
>
> Well, hmm, it's a good question:
>
> Connection information for squid:
> Number of clients accessing cache: 15283
> Number of HTTP requests received: 12646939
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 3076.3
> Average ICP messages per minute since start: 0.0
> Select loop called: 217751038 times, 1.133 ms avg
> Cache information for squid:
> Request Hit Ratios: 5min: 43.8%, 60min: 43.5%
> Byte Hit Ratios: 5min: 33.8%, 60min: 34.4%
> Request Memory Hit Ratios: 5min: 43.2%, 60min: 41.6%
> Request Disk Hit Ratios: 5min: 0.1%, 60min: 0.2%
> Storage Swap size: 0 KB
> Storage Mem size: 4194536 KB
> Mean Object Size: 0.00 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.00463 0.00463
> Cache Misses: 0.01745 0.01847
> Cache Hits: 0.00000 0.00000
> Near Hits: 0.00286 0.00286
> Not-Modified Replies: 0.00000 0.00000
> DNS Lookups: 0.00000 0.00000
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 246667.838 seconds
> CPU Time: 10025.721 seconds
> CPU Usage: 4.06%
> CPU Usage, 5 minute avg: 3.49%
> CPU Usage, 60 minute avg: 3.61%
> Process Data Segment Size via sbrk(): 92 KB
> Maximum Resident Size: 8809296 KB
> Page faults with physical i/o: 1
> Memory accounted for:
> Total accounted: 4409635 KB
> memPoolAlloc calls: 2099536771
> memPoolFree calls: 2095137519
> File descriptor usage for squid:
> Maximum number of file descriptors: 11095
> Largest file desc currently in use: 324
> Number of file desc currently in use: 309
> Files queued for open: 0
> Available number of file descriptors: 10786
> Reserved number of file descriptors: 100
> Store Disk files open: 0
> IO loop method: kqueue
> Internal Data Structures:
> 155377 StoreEntries
> 155377 StoreEntries with MemObjects
> 154959 Hot Object Cache Items
> 0 on-disk objects
>
>
> "Mean Object Size" is 0.00 KB, which must be a display error. :)
>
> If we divide "Storage Mem size" of 4194536 KB by 155377 "StoreEntries
> with MemObjects" then that would suggest mean object size of about 27
> KB, which intuitively sounds reasonable.
>
>> The amount of metadata kept per cache_mem object is pretty large and
not
>> accounted for in cache_mem limits, much larger than the amount kept for
>> disk-only objects.
>
> If there is really another 27 KB of metadata per object, then what can
> we do to reduce the problems with fork()ing new URL rewriters? As it
> is we have to double memory the requirements (again) in order to avoid
> once-a-day downtime due to log rotation.

Accepting the challenge and going of the side tangent ;) ....

redirect helpers can be reduced or removed in a lot of cases by using:

1) The deny_info bounce feature:

 # direct any request for the a.example.com domain to the front page of
b.example.com
 acl domainAbounce dstdomain a.example.com
 http_access deny domainAbounce
 deny_info http://b.example.com/ domainAbounce

NP: Squid 3.HEAD is bringing in the ability on top of this to patch a URL
together from selective parts of new text and original request. ie bounce
http://foo/file to http://foo/newstuff/file
 The patch is at
http://www.squid-cache.org/Versions/v3/HEAD/changesets/squid-3-10060.patch
for those who want to patch a 3.1* beta release instead of using the
volatile 3.2 development code.

2) Using access control lists to restrict the requests going through the
redirectors. Reducing the load on helpers, and thus how many are needed to
cope with that load.

 acl domainRewrite dstdomain foo.example.com
 url_rewrite_access allow domainRewrite
 url_rewrite_access deny all

>
> Would we be better off to declare a large memfs and have squid use it
> as "disk" storage?

Maybe. We would like to diagnose this problem and fix it properly, but if
its too much hassle you can go that way.

Amos
Received on Tue Nov 24 2009 - 01:06:12 MST