RE: [squid-users] High CPU usage and degraded service time after 2 weeks of activity

Guillaume,

Did you ever find a resolution to this issue? I'm running a very similar
config and running into very similar problems - only on more servers
using more memory and the "RHEL" squid package on CentOS 5 x86_64. Same
symptoms - no paging going on, only using 5.5G of the 8G of ram. It will
run fine for a few days. But then squid will totally consume 1 of the 4
cores in the system (two dual-core AMD Opteron(tm) Processor 2212) but
after restart only 10-20% of one core. The only significant difference
other than sizing is that I have memory_replacement_policy set at lru
instead of heap GDSF.

I haven't had the opportunity to place squid in debug mode though to see
if I get the same errors in the logs, but there's nothing fishy in
cache.log with "debug_options ALL,1 33,2" set.

Thanks,
Tony

> -----Original Message-----
> From: Guillaume Smet [mailto:guillaume.smet_at_gmail.com]
> Sent: Tuesday, February 26, 2008 2:35 PM
> To: squid-users_at_squid-cache.org
> Cc: vch_at_accelance.fr; laurent.lebreton_at_accelance.fr
> Subject: [squid-users] High CPU usage and degraded service time after
2
> weeks of activity
>
> Hi squid-users,
>
> We recently experienced a problem on our new Squid setup (2 Squid
> servers configured as reverse proxy - mostly the same configuration as
> before except we allocated more memory and disk on the new servers -
> the old boxes didn't have this problem). After 2 weeks of very good
> performances, both Squid instances have begun to use a lot of CPU
> resources (between 75 and 100% of one core instead of between 0 and
> 10%): performances started to be really bad especially during peak
> hours.
> Same number of queries/s, same hit ratio but service time was really
> degraded. We let both Squid running like that for a week and the
> situation didn't improve.
>
> We restarted both Squid servers today and it fixed the problem for
> now: service time is back to normal.
>
> We found nothing in the cache.log. We decided to run one of the
> servers with full debug for a couple of minutes to see if we could
> find useful information. During these two minutes, we have a lot of
> clientReadRequest: FD XXX: no data to process ((11) Resource
> temporarily unavailable) in the logs (20k during these 2 minutes) but
> we don't know if it can be related. Here is a bit of context around a
> Resource temporarily unavailable line.
> 2008/02/26 17:25:55| destroying entry 0x3b0e2500: 'Connection:
keep-alive'
> 2008/02/26 17:25:55| cbdataFree: 0x2aab3c2ce2a8
> 2008/02/26 17:25:55| cbdataFree: 0x2aab3c2ce2a8 has 1 locks, not
freeing
> 2008/02/26 17:25:55| clientKeepaliveNextRequest: FD 659 reading next
req
> 2008/02/26 17:25:55| commSetTimeout: FD 659 timeout 120
> 2008/02/26 17:25:55| clientReadRequest: FD 659: reading request...
> 2008/02/26 17:25:55| clientReadRequest: FD 659: no data to process
> ((11) Resource temporarily unavailable)
> 2008/02/26 17:25:55| cbdataLock: 0x2aab0bdf7418
> 2008/02/26 17:25:55| cbdataValid: 0x2aab0bdf7418
> 2008/02/26 17:25:55| cbdataUnlock: 0x2aab0bdf7418
> 2008/02/26 17:25:55| commSetSelect: FD 659 type 1
> 2008/02/26 17:25:55| commSetEvents(fd=659)
> 2008/02/26 17:25:55| cbdataUnlock: 0x2aab3c2ce2a8
>
> We also noticed that we have negative numbers in the memory
> information of the cachemgr but we don't know if it's relevant:
> Memory usage for squid via mallinfo():
> Total space in arena: -1419876 KB
> Ordinary blocks: -1420149 KB 579 blks
> Small blocks: 0 KB 0 blks
> Holding blocks: 7564 KB 8 blks
> Free Small blocks: 0 KB
> Free Ordinary blocks: 272 KB
> Total in use: -1412585 KB 100%
> Total free: 272 KB 0%
> Total size: -1412312 KB
>
> Background information:
> CentOS 5 x86_64
> Squid 2.6STABLE18
> 8GB of memory
> one Xeon E5345 @ 2.33GHz per box
> ~ 15 Mb/s per box during peak hours
> ~ 200 requests/s
>
> Cache configuration:
> cache_mem 2000 MB
> cache_dir aufs /data/services/squid/cache 8000 16 256
>
> cache_swap_low 90
> cache_swap_high 95
> cache_replacement_policy lru
> memory_replacement_policy heap GDSF
> maximum_object_size_in_memory 150 KB
>
> The setup is a reverse proxy setup with several ACLs, 2 active ports,
> 2 delay pools, ICP between both servers but nothing really fancy. I
> can provide the full squid.conf if needed.
>
> The Squid process was using approximately 3.2 GB of memory on each
box.
>
> Does anybody have any idea on how we can fix this problem or how we
> can diagnose what happens?
>
> Thanks in advance.
>
> --
> Guillaume
Received on Mon Jun 16 2008 - 17:31:48 MDT