Re: [squid-users] Object becomes STALE: refresh_pattern min and max

Hi -

I went through your same thinking as you described below.

I checked the Expires header from the server and we do not set one. I checked via Fiddler web debug tool. I also verified with the dev guys here regarding no Expires header. I have set the min and max via refresh_pattern because of the absence of the Expires header thinking that Squid would keep it FRESH.

I recently posted the details of the Squid logs, but will re-post it here so that the thread will follow properly (it may help others when search).

Hoping someone could explain the behavior I see below.

File Properties of video file (Pacific Daylight Time (PDT))

Created On: Monday, September 22, 2008, 8:59:35 AM

Modified On: Monday, September 22, 2008, 8:59:35 AM

Accessed On: Today, September 24, 2008, 3:53:12 AM

*******************************************************************
Wget Grabs File (Time in India Standard Time (IST))

--04:38:35-- http://ftp.mydomain.com/websites/data/myvideofile.vid
=> `/WGET/Temp/myvideofile.vid'
04:38:54 (93.91 KB/s) - `/WGET/Temp/myvideofile.vid' saved [1791244/1791244]

The access.log confirms initial pre-fetch by wget.

1222124934.241 18968 192.168.200.4 TCP_MISS/200 1791684 GET http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 video/jpeg

UTC = Mon, 22 Sep 2008 23:08:54 GMT

The store.log shows a write from memory to disk:

1222124934.241 SWAPOUT 00 00057B65 1E18E35BDC9307C6BC3FBEFD5B4120A3 200 1222124765 1222099175 -1 video/jpeg 1791244/1791244 GET http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Mon, 22 Sep 2008 23:08:54 GMT

*******************************************************************

Then Store.log shows release or removal from cache:

1222253725.068 RELEASE 00 00057B65 605FAC36E93B0CDE81902BBC6C5EC71A 200 1222124765 1222099175 -1 video/jpeg 1791244/-279 GET http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Wed, 24 Sep 2008 10:55:25 GMT

Notice
the -1 for expiration header (I do not set one on the object). My min
age is 5 days so I'm not sure why the object would be released from
cache in less than 2 days.

If the object was released from
cache, when the user tried to access file, Squid reports
TCP_REFRESH_MISS, which to me means that it was found in cache but when
it sends a If-Modified-Since request, it thinks that the file has been
modified (which it was not as seen by the lastmod date indicated in the
store.log below.

*******************************************************************

User accessed file (access.log):

1222253742.005 17275 192.168.200.52 TCP_REFRESH_MISS/200 1791688 GET http://ftp.mydomain.com/websites/data/myvideofile.vid - DIRECT/69.43.136.41 video/jpeg

UTC = Wed, 24 Sep 2008 10:55:42 GMT

Then store.log shows a write to disk

1222253742.005 SWAPOUT 00 00088336 1E18E35BDC9307C6BC3FBEFD5B4120A3 200 1222253575 1222099175 -1 video/jpeg 1791244/1791244 GET http://ftp.mydomain.com/websites/data/myvideofile.vid

UTC = Wed, 24 Sep 2008 10:55:42 GMT
datehdr: Wed, 24 Sep 2008 10:55:55 GMT
lastmod: Mon, 22 Sep 2008 15:59:35 GMT

----- Original Message ----
From: Michael Alger <squid_at_mm.quex.org>
To: squid-users_at_squid-cache.org
Sent: Wednesday, September 24, 2008 4:49:38 AM
Subject: Re: [squid-users] Object becomes STALE: refresh_pattern min and max

On Wed, Sep 24, 2008 at 03:39:16AM -0700, BUI18 wrote:
> I have squid box with tons of disk for the cache_dir
> (hundreds of GB). I use wget to perform some pre-fetching of large
> video files. I've set the min and max age to 5 days and 7 days (in
> minutes). And although I have plenty of disk space available, I still
> receive TCP_REFRESH_MISS for files that had been pre-fetched and later
> accessed the same day. Does anyone know why Squid would consider it as
> STALE? I thought that by setting the min value for refresh_pattern for
> the video file would guarantee freshness. Not only does the cache
> consider it STALE, it then goes and pre-fetches a new copy even though
> I know that the video file has not changed. Any help would be greatly
> appreciated. Thanks.

The fact that it's doing TCP_REFRESH_xxx means squid does have a
cached copy which it considers potentially stale. So it's sending an
If-Modified-Since request to the origin server. The origin is then
either saying "yes, it's been modified since you retrieved it --
here's a new one"; or it has no idea how to handle IMS and is
sending the whole object regardless.

What Expires: header is the server sending? You can use the -S
switch with wget to show the server response headers when you're
doing your pre-fetch. If they look fine, maybe keep a copy of them
and compare later in the day when squid decides it needs a new one.

I assume you're setting the min and max age via refresh_pattern
lines? Remember that these are only used in the absence of an
Expires header.

      
Received on Wed Sep 24 2008 - 12:29:59 MDT