[squid-users] problem: remote site times out, provider blames squid proxy

Hello,

I’m the proxy administrator for a university. We’re running squid 2.6-stable 21 on RHEL.

Earlier this year one of the departments purchased a “customer help” service from RightNow ( www.rightnow.com ) who then created a site for our university. It is at the same IP address as many other RightNow sites, I believe they use a virtual IP with a load balancer and a heap of sites behind it. The IP address for our site has over 200 other customer sites associated with it.

http://latrobe.custhelp.com

The problem we have is that this site often fails to load via our squid proxies, clicking on links on the front page – specifically different “answers” – one can wait a long time for a response. The site does load fine if going direct. After much too-ing and fro-ing the tech support at RightNow maintain that our proxy server is displaying aberrant behavior. This is puzzling as we are not a small university and our 35,000 users of the web don’t have this problem with any site except the RightNow site. Our proxy systems handle millions of requests just fine.

Most other RightNow sites at the same IP address do load correctly. Ours often times out. This is an intermittent problem, there are rare days when the site always loads fine. I have taken many TCPdumps of connections from proxy requests to the site in question. I interpret the dumps as showing that during time-outs after 60 seconds (the timeout value I have set the proxy to) the proxy gives up, re-sets the connection and tries again. Sometimes this is repeated until the proxy gives the user the usual “site is down or busy” error message, sometimes eventually the site loads OK.

RightNow claim that data starts coming back from their server but then the proxy starts sending other packets to their server, which their server responds to but our proxy server does not.

Normally I would just add a line to proxy.pac specifying all connections to the site in question should go direct but our gateways have port 80 blocked, only the proxies can go out on port 80.

We have experienced identical behavior now with no less than 7 of our proxy servers. I have tried with a test proxy server with the bare minimum of configuration also, practically the default config. What is difficult to understand is how one site in particular has this issue, most other RightNow customer sites always load fine although I have found one or two at the same IP address that have the same issue as us.

In over 10 years of squid use we’ve never had such a long standing or intractable issue. The RightNow techs are adamant the problem is our proxies. Is this a unique error, have they created a site that has discovered a bug in squid I didn’t know about? Has anyone else seen a problem like this? Any tips or ideas of what I should look into?

cheers

Howard Cock

Senior Unix Engineer
Information and Communications Technology
La Trobe University, 3086
Received on Sun Nov 15 2009 - 23:21:38 MST