RE: [squid-users] squid2.7 Stable 6 passes in intervals where cpu usage hits 100%

Hi Amos,

Thanks for the info.

According to what you have mentioned below what do you think should be the
value of cache_swap_low and high values to lower the load on the system
although the hardware is big enough I think to hold such a load (all system
us SAS based with 15k rpm drives and has 16GB of pyshical memory.)
 
>That indicates that when the cache dir gets more than 95% full it needs
>to clear at least 2GB of objects (2% of 115,000MB).
>
>Depending on your disks and avg object sizes this may be a large part of
>the load.

As for the regex acls since like I have said these specified by the
videocache python squid plugin redirector setup I can try turning all of
these off and see if the problem persists or no.
And these regex acls are used only with below access lines

url_rewrite_access deny videocache_deny_url
url_rewrite_access allow videocache_allow_url
url_rewrite_access allow videocache_allow_dom

>A lot of those can be reduced to dstdomain or at least dstdomain
>followed by a regex.
>
>ie:
>acl A dstdomain .googlevideo.com
>acl R urlpath_regex -i /videoplayback
>
>http_access allow/deny A R
>
>Using dstdomain (very fast test) at the start of an access line to
>determine if the slower regex later on in the line is even needed can
>produce a dramatic speed boost in response times.
>
>Where you are testing for file types the rep_mime_type on reply can
>check the mime faster than a regex can test the entire URL.
>
>Also matching all of that file type by mime from a whole dstdomain can
>be more effective and faster than a specific regex testing for a range
>of specific sub-domains and file endings.
>
>Also you do not indicate which if the *_access lines these are being
>tested in. It's a very good idea to only to them in one of either
>http_access or http_reply_access, not both.

Ragheb Rustom

-----Original Message-----
From: Amos Jeffries [mailto:squid3_at_treenet.co.nz]
Sent: Saturday, May 30, 2009 11:16 AM
To: Ragheb Rustom
Cc: squid-users_at_squid-cache.org; crobertson_at_gci.net
Subject: Re: [squid-users] squid2.7 Stable 6 passes in intervals where cpu
usage hits 100%

Ragheb Rustom wrote:
> Hi Chris,
>
> Thank you for the reply. I will use squid -k debug to check what is going
> on.
>
> As for the additional info sorry for not clearing the squid setup more
> anyway this is squid configuration values you have guessed as taken from
my
> squid.conf
>
> Wild guesses based on the information given:
>
> *) You are using UFS for your cache_dir instead of AUFS.
>
> cache_dir aufs /cache1 115000 16 256
> cache_dir aufs /cache2 115000 16 256
>
> I am using AUFS not UFS.
>
> *) Your cache is pretty large and your cache_swap_high and
> cache_swap_low are still at the defaults
>
> cache_swap_low 93
> cache_swap_high 95
>
> Do u think these values are reasonable enough regarding the above values
of
> my Cacche dir or should they be changed.
>

That indicates that when the cache dir gets more than 95% full it needs
to clear at least 2GB of objects (2% of 115,000MB).

Depending on your disks and avg object sizes this may be a large part of
the load.

> *) You are using large lists of regex ACLs
>
> acl videocache_allow_url url_regex -i \.youtube\.com\/get_video\?
> acl videocache_allow_url url_regex -i \.googlevideo\.com\/videoplayback
> \.googlevideo\.com\/get_video\?
> acl videocache_allow_url url_regex -i \.google\.com\/videoplayback
> \.google\.com\/get_video\?
> acl videocache_allow_url url_regex -i \.google\.[a-z][a-z]\/videoplayback
> \.google\.[a-z][a-z]\/get_video\?
> acl videocache_allow_url url_regex -i
>
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]
>
?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][
> 0-9]?)\/videoplayback\?
> acl videocache_allow_url url_regex -i
>
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]
>
?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][
> 0-9]?)\/get_video\?
> acl videocache_allow_url url_regex -i
> proxy[a-z0-9\-][a-z0-9][a-z0-9][a-z0-9]?\.dailymotion\.com\/
> acl videocache_allow_url url_regex -i vid\.akm\.dailymotion\.com\/
> acl videocache_allow_url url_regex -i
> [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
> acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
> acl videocache_allow_url url_regex -i
> va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
> acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
> acl videocache_allow_url url_regex -i
\.msn\.com\.edgesuite\.net\/(.*)\.flv
> acl videocache_allow_url url_regex -i
> media[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
> mobile[a-z0-9]?[a-z0-9]?[a-z0-9]?\.tube8\.com\/
> acl videocache_allow_url url_regex -i \.mais\.uol\.com\.br\/(.*)\.flv
> acl videocache_allow_url url_regex -i
>
\.video[a-z0-9]?[a-z0-9]?\.blip\.tv\/(.*)\.(flv|avi|mov|mp3|m4v|mp4|wmv|rm|r
> am)
> acl videocache_allow_url url_regex -i video\.break\.com\/(.*)\.(flv|mp4)
> acl videocache_allow_dom dstdomain v.mccont.com dl.redtube.com
> .cdn.dailymotion.com
> acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com
> http:\/\/www\.youtube\.com
>
> acl bad_requests urlpath_regex -i cmd.exe \/bin\/sh default\.ida?XXXXXXX
> omg.pif
>
> acl music url_regex -i .wav
> acl video url_regex -i .avi .mpe
> acl cddown url_regex -i .iso .raw
>
> These are all the acl regex entries I use in my squid.conf file mainly
most
> of them are those of videocache.

A lot of those can be reduced to dstdomain or at least dstdomain
followed by a regex.

ie:
acl A dstdomain .googlevideo.com
acl R urlpath_regex -i /videoplayback

http_access allow/deny A R

Using dstdomain (very fast test) at the start of an access line to
determine if the slower regex later on in the line is even needed can
produce a dramatic speed boost in response times.

Where you are testing for file types the rep_mime_type on reply can
check the mime faster than a regex can test the entire URL.

Also matching all of that file type by mime from a whole dstdomain can
be more effective and faster than a specific regex testing for a range
of specific sub-domains and file endings.

Also you do not indicate which if the *_access lines these are being
tested in. It's a very good idea to only to them in one of either
http_access or http_reply_access, not both.

>
> I will use the squid -k debug and try and get more info of what is going
on
> during the high cpu usage period and will share whatever I find on
> squid-users mail list.
>
> Thanks Chris for your support.
>
> Ragheb
>
> -----Original Message-----
> From: crobertson_at_gci.net [mailto:crobertson_at_gci.net]
> Sent: Friday, May 29, 2009 10:18 PM
> To: squid-users_at_squid-cache.org
> Subject: Re: [squid-users] squid2.7 Stable 6 passes in intervals where cpu
> usage hits 100%
>
> Ragheb Rustom wrote:
>> Dear All,
>>
>> I have been noticing for some time now that my squid server passes in
> nearly
>> regular intervals during the day where the squid process CPU usage hits
> 100%
>> for around 20 - 30 seconds or so during which squid will not serve
> requests.
>> After these 20 -30sec the squid CPU usage drops to around 5 - 13% and
>> website navigation is normal again. This happens like I said before at
>> nearly regular intervals during each day several times and it happens
only
>> for around 20 -30sec. I have been observing the cache.log file but there
> is
>> nothing apparent there. Is there a way that I know what squid is doing
> when
>> this problem happens to try and resolve it as trying to solve it like it
> is
>> now with no info about what is going on is somewhat vague.
>
> squid -k debug
>
>> I think this
>> problem happened after I have changed memory and disk replacement
policies
>> from lru to heap but I am not so sure that this is the source of the
>> problem. I am using squid2.7-Stable6 on a fedora9 server which is a dual
>> xeon quad 2.8Ghz CPU with 16Gb of Ram installed and 3 SAS hard drives.
One
>> more thing this server is also running videocache 1.9.1 but it has been
>> running videocache also before the problem started with no problems
>> whatsoever. I appreciate any help you can offer me.
>>
>
> Wild guesses based on the information given:
>
> *) You are using UFS for your cache_dir instead of AUFS.
> *) Your cache is pretty large and your cache_swap_high and
> cache_swap_low are still at the defaults
> *) You are using large lists of regex ACLs
>
>> Sincerely,
>>
>> Ragheb Rustom
>> Smartelecom S.A.R.L
>> Sin el fil - Mar Elias Street
>> Absi Center - Ground Floor
>> Beirut, Lebanon
>> Email: ragheb_at_smartelecom.org
>>
>
> Chris
>
>

-- 
Please be using
   Current Stable Squid 2.7.STABLE6 or 3.0.STABLE15
   Current Beta Squid 3.1.0.8 or 3.0.STABLE16-RC1