Re: [squid-users] Squid eating too much memory

Hi Rihad,

rihad wrote:
> Tek Bahadur Limbu wrote:
>> Hi Rihad,
>>
>> rihad wrote:
>>> CacheMgr output:
>>> Memory accounted for:
>>> Total accounted: 1323944 KB
>>>
>>> Memory usage using top(1):
>>> PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU
>>> COMMAND
>>> 29601 squid 29 20 0 2533M 2465M kserel 92:12 0.00% squid
>>>
>>>
>>> Almost twice as much memory! Any hints?
>>
>> What is your settings for the following parameters?
>>
>> cache_mem
>> maximum_object_size
>> maximum_object_size_in_memory
>> cache_replacement_policy
>> memory_replacement_policy
>> ipcache_size
>> fqdncache_size
>>
> cache_mem 1000 MB
> maximum_object_size 100 MB
> # maximum_object_size_in_memory 8 KB
> cache_replacement_policy heap LFUDA
> memory_replacement_policy heap LFUDA
> # ipcache_size 1024
> # fqdncache_size 1024

I also suggest you to reduce the size of your cache_mem.

>
>> Which storage scheme are you using?
>>
> aufs
>

Should work nicely on a 6.2

>>>
>>> Number of clients accessing cache: 517
>>> Normally load is within 500-1500 clients.
>>
>> How long is your Squid process been running?
>>
> 3 to 4 days, but it doesn't matter: I've had weeks or months of uptime,
> too. Eventually the process hits its memory size limit (kern.maxdsiz)
> set at 2.5 gigs at the moment and croaks (and restarts) unless I
> decrease cache_mem. There's little point in using less RAM as the box
> has ~3.3 gigs of it, and is dedicated to Squid (and its dedicated
> dnscache).

You can also try increasing your kern.maxdsiz then.

>
>> Your server's memory usage seems unusually high even for 1500 clients.
>> My proxy server which serves 2000-3000 clients is currently using 650
>> MB of memory despite running for more than 55 days.
>>
> How big is your cache store?

I have a COSS store with a size of 10 GB. Used to have bigger cache
sizes, but COSS rebuilding time is it's weakest point in my opinion.

>
>>
>>>
>>> Squid 2.6.16
>>> FreeBSD 6.2-RELEASE-p8
>>
>> What is your machines specs?
>>
> # grep -e ^CPU: -e memory -e ^ad /var/run/dmesg.boot
> CPU: Intel(R) Pentium(R) D CPU 2.80GHz (2813.85-MHz 686-class CPU)
> real memory = 3489595392 (3327 MB)
> avail memory = 3414970368 (3256 MB)
> ad4: 238475MB <Seagate ST3250824AS 3.AAE> at ata2-master SATA150
> ad6: 238475MB <Seagate ST3250824AS 3.AAE> at ata3-master SATA150
>

Pretty decent hardware.

> Two cache_dir's lie on both disks (25 gigs each at the moment, but I'm
> planning for more).

The bigger the cache sizes, the more rebuilding time it takes!

>
>>>
>>> Port compiled with all options unchecked and both AUFS & KQUEUE checked
>>> (as per "make config" and /var/db/ports/squid/options)
>>
>> Can you post the full output of "squidclient mgr:info"?
>>
> Squid Object Cache: Version 2.6.STABLE16
> Start Time: Thu, 18 Oct 2007 17:10:09 GMT
> Current Time: Sun, 21 Oct 2007 09:39:51 GMT
> Connection information for squid:
> Number of clients accessing cache: 517
> Number of HTTP requests received: 9410288
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 2431.8
> Average ICP messages per minute since start: 0.0
> Select loop called: 74113837 times, 3.133 ms avg
> Cache information for squid:
> Request Hit Ratios: 5min: 35.6%, 60min: 37.1%
> Byte Hit Ratios: 5min: 21.4%, 60min: 18.1%
> Request Memory Hit Ratios: 5min: 12.0%, 60min: 15.0%
> Request Disk Hit Ratios: 5min: 22.9%, 60min: 26.2%
> Storage Swap size: 22427274 KB
> Storage Mem size: 1023900 KB
> Mean Object Size: 13.62 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.08265 0.08265
> Cache Misses: 0.17711 0.18699
> Cache Hits: 0.00463 0.00562
> Near Hits: 0.12106 0.14252
> Not-Modified Replies: 0.00286 0.00379
> DNS Lookups: 0.11405 0.10906
> ICP Queries: 0.00000 0.00000

You seem to have a fast connection? Probably fiber optic? How big is
your bandwidth pipe?

Still, your DNS lookups seems a little slow compared to your Median
response time.

You also mentioned that you are running a DNS caching name server on
this squid box?

So you are just using a single squid box?

> Resource usage for squid:
> UP Time: 232181.468 seconds
> CPU Time: 5971.253 seconds
> CPU Usage: 2.57%
> CPU Usage, 5 minute avg: 3.98%
> CPU Usage, 60 minute avg: 3.55%
> Process Data Segment Size via sbrk(): -1583756 KB
> Maximum Resident Size: 2549632 KB
> Page faults with physical i/o: 3799

Very little CPU utilization. That's one of the best feature of Squid-2.6
I guess. Are you having service outages? Your page faults seems a little
high for a cache which is only 3-4 days old.

> Memory accounted for:
> Total accounted: 1333488 KB
> memPoolAlloc calls: 1090270339
> memPoolFree calls: 1082572188
> File descriptor usage for squid:
> Maximum number of file descriptors: 11072
> Largest file desc currently in use: 1715
> Number of file desc currently in use: 1622
> Files queued for open: 0
> Available number of file descriptors: 9450
> Reserved number of file descriptors: 100
> Store Disk files open: 41
> IO loop method: kqueue
> Internal Data Structures:
> 1652462 StoreEntries
> 200285 StoreEntries with MemObjects
> 200125 Hot Object Cache Items
> 1646504 on-disk objects
>

By the way, just curious, are you an internet service provider?

>
>>
>> Thanking you...
>>
> Thank you too!

Thanking you...

>
>
>

-- 
With best regards and good wishes,
Yours sincerely,
Tek Bahadur Limbu
System Administrator
(TAG/TDG Group)
Jwl Systems Department
Worldlink Communications Pvt. Ltd.
Jawalakhel, Nepal
http://www.wlink.com.np
http://teklimbu.wordpress.com