Re: [squid-users] Re: can squid load data into cache faster than sending it out?

On Sat, May 07, 2011 at 02:32:22PM +1200, Amos Jeffries wrote:
> On 07/05/11 08:54, Dave Dykstra wrote:
> >Ah, but as explained here
> > http://www.squid-cache.org/mail-archive/squid-users/200903/0509.html
> >this does risk using up a lot of memory because squid keeps all of the
> >read-ahead data in memory. I don't see a reason why it couldn't instead
> >write it all out to the disk cache as normal and then read it back from
> >there as needed. Is there some way to do that currently? If not,
>
> Squid should be writing to the cache in parallel to the data
> arrival, the only bit required in memory being the bit queued for
> sending to the client. Which gets bigger, and bigger... up to the
> read_ahead_gap limit.

Amos,

Yes, it makes sense that it's writing to the disk cache in parallel, but
what I'm asking for is a way to get squid to keep reading from the
origin server as fast as it can without reserving all that memory. I'm
asking for an option to not block the reading from the origin server &
writing to the cache when the read_ahead_gap is full, and instead read
data back from the cache to write it out when the client is ready for
more. Most likely the data will still be in the filesystem cache so it
will be fast.

> IIRC it is supposed to be taken out of the cache_mem space
> available, but I've not seen anything to confirm that.

I'm sure that's not the case, because I have been able to force the
memory usage to grow by more than the cache_mem setting by doing a
number of wgets of largeish requests in parallel using --limit-rate to
cause them to take a long time. Besides, it doesn't make sense that it
would do that when the read_ahead_gap is far greater than
maximum_object_size_in_memory.

> >perhaps I'll just submit a ticket as a feature request. I *think* that
> >under normal circumstances in my application squid won't run out of
> >memory, but I'll see after running it in production for a while.

So far I haven't seen a problem but I can imagine ways that it could
cause too much growth so I'm worried that one day it will.

- Dave
Received on Tue May 10 2011 - 16:34:32 MDT