RE: [squid-users] Re: Purging cached web objects

I'm looking at writing a web based admin application for squid and have
been using the cache manager functions to help retrieve what's stored in
the cache. With Henrik's help I've discovered that in memory objects are
stored in both the "vm_objects" & "objects" ("objects" lists both in
memory and on disk) functions, they are listed with their urls, objects
stored on disk are also listed, but not with their urls. If the cache
manager "objects" function could be extended to also list the urls it
would probably help. You'd be able to make an http request to the proxy
for "cache_object://localhost/objects" and do your own regexing to find
the urls you're after and send purge requests for those. It wouldn't be
the most efficient way of doing it, but it'd be a start!

Joe

Joe Tiedeman
Support Analyst
Higher Education Statistics Agency (HESA)
95 Promenade, Cheltenham, Gloucestershire GL50 1HZ
T 01242 211167 F 01242 211122 W www.hesa.ac.uk

-----Original Message-----
From: Chris Woodfield [mailto:rekoil_at_semihuman.com]
Sent: Monday 1 September 2008 16:17
To: Amos Jeffries
Cc: RW; squid-users_at_squid-cache.org
Subject: Re: [squid-users] Re: Purging cached web objects

squidpurge works, but it's hardly ideal, especially on squids with big
disks...in my testing on a box with 3x1TB cache_dirs, it took 15 minutes
to run and thrashed the disks pretty hard while it was running,
affecting response time for production traffic.

The reason for this is that squid stores each object record internally
as a hash, not the URL itself, which means that in order to search for
regex matches, it's necessary to look at every file in every cache_dir
to check against the regex.

An easily-searchable URL datastore would help immensely here. As a mad
experiment a while back, a former colleague hacked a SQL update into the
store and release functions, but it's unlikely anything like that would
work well in production without some serious work to guarantee squid/DB
data integrity.

Some sort of internal b-tree that stores all currently-cached URLs might
be a solution...or even an internal sqlite implementation? Has anyone
else ever proposed such a solution?

-C

On Aug 25, 2008, at 11:55 PM, Amos Jeffries wrote:

>> On Sun, 24 Aug 2008 08:59:07 +1200
>> Amos Jeffries <squid3_at_treenet.co.nz> wrote:
>>
>>> Paras Fadte wrote:
>>>> Hi,
>>>>
>>>> Is there any utility for purging cached web objects in squid with
>>>> wildcard support ?
>>>
>>> Not that we know of.
>>
>> You presumable know about squidpurge. Has it broken or something?
>>
>> http://www.wa.apana.org.au/~dean/squidpurge/
>>
>
> Ah. no. I didn't. I see the name fly by every now and again, but
> haven't really noticed it.
>
> Amos
>
>

______________________________________________________________

This incoming email was virus scanned for HESA by MessageLabs.
______________________________________________________________

_____________________________________________________________________

Higher Education Statistics Agency Ltd (HESA) is a company limited by
guarantee, registered in England at 95 Promenade Cheltenham GL50 1HZ.
Registered No. 2766993. The members are Universities UK and GuildHE.
Registered Charity No. 1039709. Certified to ISO 9001 and BS 7799.
 
HESA Services Ltd (HSL) is a wholly owned subsidiary of HESA,
registered in England at the same address. Registered No. 3109219.
_____________________________________________________________________

This outgoing email was virus scanned for HESA by MessageLabs.
_____________________________________________________________________
Received on Mon Sep 01 2008 - 15:47:51 MDT