Re: [squid-users] limit squid memory ram use - squid becomes slow when ram full

On 26/04/11 23:06, rpereyra_at_lavabit.com wrote:
>> On Mon, 11 Apr 2011 07:52:41 -0400 (EDT), rpereyra_at_lavabit.com wrote:
>>> Hi
>>>
>>> How I can limit the ram memory use in my squid/tproxy box ?
>>>
>>> I have a fast server with 16Gb ram. The average bandwidth is about
>>> 60-70
>>> Mb/s.
>>>
>>> The bridge works well but when the cache and memory becomes full its
>>> goes
>>> slow and becomes unusable.
>>>
>>> The cache is 10G size.
>>>
>>> I see that a few hours to be working and have used the 16 GB of RAM
>>> starts to run slow.
>>>
>>> Any help ?. I have configured some memory optimization options but
>>> looks
>>> don't help for me.
>>>
>>> Thanks in advance
>>>
>>> roberto
>>>
>>> This is my config:
>>>
>>> -------------------------------------
>>>
>>> cache_mem 10 MB
>>> memory_pools off
>>> cache_swap_low 94
>>> cache_swap_high 95
>>>
>> <snip>
>>> # Example rule allowing access from your local networks.
>>> # Adapt to list your (internal) IP networks from where browsing
>>> # should be allowed
>>> acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
>>> acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
>>> acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
>>> acl localnet src fc00::/7 # RFC 4193 local private network range
>>> acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged)
>>> machines
>>>
>>> acl net-g1 src 200.117.xxx.xxx/24
>>> acl net-g2 src 200.xxx.xxx.xxx/24
>>> acl net-g3 src 190.xxx.xxx.xxx/24
>>>
>> <snip>
>>>
>>> http_access allow net-g1
>>> http_access allow net-g2
>>> http_access allow net-g3
>>
>> Being "allow" you can grab some extra speed by combining these all
>> under one ACL name. "localnet" being the standard one for local network
>> ranges.
>>
>> <snip>
>>>
>>> # Uncomment and adjust the following to add a disk cache directory.
>>> cache_dir ufs /var/spool/squid 10000 64 256
>>>
>> <snip>
>>
>>
>> Given those cache sizes, your Squid box should be using around 110 MB
>> of RAM for index plus a little. Even assuming a worst-case of a minutes
>> traffic accumulated in transit buffers comes nowhere close to filling 16
>> GB up.
>>
>> Some questions that may help narrow down where the slow is coming from:
>> What version of Squid is this?
>> What are your avg object size?
>> How many concurrent client connections?
>> "slow" and "normal" response speeds?
>> Do you notice any change in the Squid->Internet request types during
>> the slowdown? (ie a move to extra MISS/HIT/IMS/REFRESH)
>>
>> And like Michael said the disk IO stats are important to look at. When
>> the cache_dir gets to 94% full it will start spending CPU and disk
>> cycles on erasing objects. If it reaches 95% a larger portion of cycles
>> get used until it drops down below 94% again.
>>
>>
>> Amos
>>
>
> Hi all !
>
> These are two test samples when the proxy becomes slow.
>
> Any clue ? Maybe an I/O disk issue.
>
> roberto
>
> First sample:
>
> [jorge_at_proxy ~]$ squidclient mgr:info
> HTTP/1.0 200 OK
> Server: squid/3.1.1
> Mime-Version: 1.0
> Date: Wed, 20 Apr 2011 17:38:35 GMT
> Content-Type: text/plain
> Expires: Wed, 20 Apr 2011 17:38:35 GMT
> Last-Modified: Wed, 20 Apr 2011 17:38:35 GMT
> X-Cache: MISS from proxy.net-g.com.ar
> X-Cache-Lookup: MISS from proxy.net-g.com.ar:3128
> Via: 1.0 proxy.net-g.com.ar (squid/3.1.1)
> Proxy-Connection: close
>
> Squid Object Cache: Version 3.1.1

Aha. 3.1.1. Memory leaks and Squid holding unnecessary references to
objects when they should be erased are two big performance hogs which
have been fixed since that release.
  You should find any of the 3.1.10 and later releases much better.
3.1.12 is the latest stable we have available.

> Start Time: Wed, 20 Apr 2011 17:34:35 GMT
> Current Time: Wed, 20 Apr 2011 17:38:35 GMT
> Connection information for squid:
> Number of clients accessing cache: 171
> Number of HTTP requests received: 99367
> Number of ICP messages received: 0
> Number of ICP messages sent: 0
> Number of queued ICP replies: 0
> Number of HTCP messages received: 0
> Number of HTCP messages sent: 0
> Request failure ratio: 0.00
> Average HTTP requests per minute since start: 24781.1
> Average ICP messages per minute since start: 0.0
> Select loop called: 588320 times, 0.409 ms avg
> Cache information for squid:
> Hits as % of all requests: 5min: 27.8%, 60min: 27.8%
> Hits as % of bytes sent: 5min: 16.9%, 60min: 16.9%
> Memory hits as % of hit requests: 5min: 6.3%, 60min: 6.3%
> Disk hits as % of hit requests: 5min: 45.5%, 60min: 45.5%
> Storage Swap size: 6029640 KB
> Storage Swap capacity: 98.1% used, 1.9% free

This 98.1% is the percentage of disk space filled with objects. Like I
said earlier. Once the cache gets over 95% the disk I/O and CPU cycles
get spent on garbage collection instead of handling traffic until the
usage drops below 94% again.
  This is likely a major cause of slow down.

> Storage Mem size: 319540 KB
> Storage Mem capacity: 15.4% used, 84.6% free

Huge amount of disk space used, yet only a third of the available fast
memory cache space.
  Check that your maximum_object_size_in_memory is at least bigger than
the Mean Object size indicated below...

> Mean Object Size: 14.94 KB
> Requests given to unlinkd: 0
> Median Service Times (seconds) 5 min 60 min:
> HTTP Requests (All): 0.30459 0.30459
> Cache Misses: 0.42149 0.42149
> Cache Hits: 0.00179 0.00179
> Near Hits: 0.18699 0.18699
> Not-Modified Replies: 0.00091 0.00091
> DNS Lookups: 0.03374 0.03374
> ICP Queries: 0.00000 0.00000
> Resource usage for squid:
> UP Time: 240.588 seconds
> CPU Time: 165.222 seconds
> CPU Usage: 68.67%
> CPU Usage, 5 minute avg: 67.92%
> CPU Usage, 60 minute avg: 67.92%
> Process Data Segment Size via sbrk(): 259836 KB
> Maximum Resident Size: 0 KB
> Page faults with physical i/o: 0
> Memory usage for squid via mallinfo():
> Total space in arena: 260112 KB
> Ordinary blocks: 259914 KB 946 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 313188 KB 2048 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 197 KB
> Total in use: 573102 KB 100%
> Total free: 197 KB 0%
> Total size: 573300 KB
> Memory accounted for:
> Total accounted: 517250 KB 90%
> memPool accounted: 517250 KB 90%
> memPool unaccounted: 56049 KB 10%
> memPoolAlloc calls: 42349480
> memPoolFree calls: 40063448
> File descriptor usage for squid:
> Maximum number of file descriptors: 16384
> Largest file desc currently in use: 13418
> Number of file desc currently in use: 13197

~6.5K active connections (up to 2 FD per connection).

> Files queued for open: 6
> Available number of file descriptors: 3181
> Reserved number of file descriptors: 100
> Store Disk files open: 171
> Internal Data Structures:
> 404884 StoreEntries
> 24853 StoreEntries with MemObjects
> 23380 Hot Object Cache Items
> 403505 on-disk objects
>
>
> --------------------------
>
> Second sample:
<snip>
>
> Squid Object Cache: Version 3.1.1
> Start Time: Thu, 21 Apr 2011 17:44:32 GMT
> Current Time: Thu, 21 Apr 2011 17:53:35 GMT
> Connection information for squid:
<snip>
> Internal Data Structures:
> 387582 StoreEntries
> 58603 StoreEntries with MemObjects
> 57148 Hot Object Cache Items
> 386166 on-disk objects
> [jorge_at_proxy ~]$
>

So in 6 minutes it has managed to purge ~17,300 objects from the cache
and bump ~34,000 into the high speed RAM area. Not too shabby when
handling ~6K client requests at the same time.

Overall I think you need to upgrade to 3.1.10 or later (3.1.12
preferably). See about increasing the FD available to Squid (to 24K or
so). Also convert to the faster AUFS cache_dir type.

Those changes will all impact on disk usage. After doing them re-assess
the problem see if you still need either more space or faster disks.

Amos

-- 
Please be using
   Current Stable Squid 2.7.STABLE9 or 3.1.12
   Beta testers wanted for 3.2.0.7 and 3.1.12.1