Re: [squid-users] "Quadruple" memory usage with squid

On Sun, Nov 22, 2009 at 9:03 PM, Henrik Nordstrom
<henrik_at_henriknordstrom.net> wrote:
> What is your average object size?

Well, hmm, it's a good question:

Connection information for squid:
        Number of clients accessing cache: 15283
        Number of HTTP requests received: 12646939
        Number of ICP messages received: 0
        Number of ICP messages sent: 0
        Number of queued ICP replies: 0
        Request failure ratio: 0.00
        Average HTTP requests per minute since start: 3076.3
        Average ICP messages per minute since start: 0.0
        Select loop called: 217751038 times, 1.133 ms avg
Cache information for squid:
        Request Hit Ratios: 5min: 43.8%, 60min: 43.5%
        Byte Hit Ratios: 5min: 33.8%, 60min: 34.4%
        Request Memory Hit Ratios: 5min: 43.2%, 60min: 41.6%
        Request Disk Hit Ratios: 5min: 0.1%, 60min: 0.2%
        Storage Swap size: 0 KB
        Storage Mem size: 4194536 KB
        Mean Object Size: 0.00 KB
        Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
        HTTP Requests (All): 0.00463 0.00463
        Cache Misses: 0.01745 0.01847
        Cache Hits: 0.00000 0.00000
        Near Hits: 0.00286 0.00286
        Not-Modified Replies: 0.00000 0.00000
        DNS Lookups: 0.00000 0.00000
        ICP Queries: 0.00000 0.00000
Resource usage for squid:
        UP Time: 246667.838 seconds
        CPU Time: 10025.721 seconds
        CPU Usage: 4.06%
        CPU Usage, 5 minute avg: 3.49%
        CPU Usage, 60 minute avg: 3.61%
        Process Data Segment Size via sbrk(): 92 KB
        Maximum Resident Size: 8809296 KB
        Page faults with physical i/o: 1
Memory accounted for:
        Total accounted: 4409635 KB
        memPoolAlloc calls: 2099536771
        memPoolFree calls: 2095137519
File descriptor usage for squid:
        Maximum number of file descriptors: 11095
        Largest file desc currently in use: 324
        Number of file desc currently in use: 309
        Files queued for open: 0
        Available number of file descriptors: 10786
        Reserved number of file descriptors: 100
        Store Disk files open: 0
        IO loop method: kqueue
Internal Data Structures:
        155377 StoreEntries
        155377 StoreEntries with MemObjects
        154959 Hot Object Cache Items
             0 on-disk objects

"Mean Object Size" is 0.00 KB, which must be a display error. :)

If we divide "Storage Mem size" of 4194536 KB by 155377 "StoreEntries
with MemObjects" then that would suggest mean object size of about 27
KB, which intuitively sounds reasonable.

> The amount of metadata kept per cache_mem object is pretty large and not
> accounted for in cache_mem limits, much larger than the amount kept for
> disk-only objects.

If there is really another 27 KB of metadata per object, then what can
we do to reduce the problems with fork()ing new URL rewriters? As it
is we have to double memory the requirements (again) in order to avoid
once-a-day downtime due to log rotation.

Would we be better off to declare a large memfs and have squid use it
as "disk" storage?

Thanks!
Received on Mon Nov 23 2009 - 14:22:51 MST