Re: [squid-users] -k reconfigure to slow

On Wed, 04 Nov 2009 20:34:51 -0300, Guido Marino Lorenzutti
<glorenzutti_at_jusbaires.gov.ar> wrote:
> Imagine that!
> "... Well people, i have to change a mail address in the whitelist of
> the mailserver, everyone STOP SENDING MAILS that the server needs to
> be restarted.. and ALL the childs... so.. HOLD ON THE SEND BUTTON... "
> (WFT?!)
>
> I have to apply all the changes at midnight... With one, or two
> minutes of downtime in the squid I lost a lot of connections to my
> webaps and external pages...
>
> I can measure the problem with iptables, but Im sure that at least I
> lost 300 requests in that time.
>
> And if you use a reverse proxy.. well.. prepare to cry..
>
> It would be great to have a tool to tests urls with a set of acls, so
> you can test before apply.
>
> Jorge Armando Medina <jmedina_at_e-compugraf.com> escribió:
>
>> Guido Marino Lorenzutti wrote:
>>> I don't understand why it works like this! But it seems that this is
>>> like the squid works... maybe we should also ask why the childs needs
>>> to be restarted, if you change an acl... this sucks.
>>>
>> I agree, I have same problem, whenever I add a ACL I have to
>> reconfigure, and for a few seconds users can't access the proxy, then
>> they have to reload the page :(.

We know it's a problem, not the biggest caused by reconf, and plan on
fixing it. http://wiki.squid-cache.org/Features/HotConf

The issue is that Squid currently only has _one_ configuration/setup and
there is no way before parsing the file into it to know whether some ACL
name has been changed, a listening port, or if the helper binary path has
changed. Quite nasty.

Plans are well underway to break that config up into component chunks.
pending that to have each component only restart/reconf if its details have
actually changed. Patience please, or sponsorship money to pay for faster
work on the restructuring SourceLayout and HotConf 'features'.

Amos

>>> I don't remember any other app with this popularity that works like
>>> this...
>>>
>>> instead of working better with 64bits, it works worst? I have many
>>> squids, I don't recall having one in 32bits to compare...
>>>
>>> Malte Schröder <maltesch_at_gmx.de> escribió:
>>>
>>>> Hello,
>>>> I also have one thing to add: I think it is worse on 64bit than on
>>>> 32bit
>>>> linux. My guess is that closing and forking all those processes is
just
>>>> too expansive.
>>>>
>>>>
>>>> On Wed, 04 Nov 2009 16:39:08 -0300
>>>> Guido Marino Lorenzutti <glorenzutti_at_jusbaires.gov.ar> wrote:
>>>>
>>>>> Sorry, can't help you with the problem. But I have a question
>>>>> instead :)
>>>>>
>>>>> Where did you get the ip-user-helper.pl ?
>>>>>
>>>>> Tryin' to help... i have also 1k connections, squid 2.7 and it takes
>>>>> that time too... i found that if i reduce the number of helpers it
>>>>> starts up faster.
>>>>>
>>>>> I have 180 NTLM, 10 basic, and 75 squid_ldap_group childrens..
>>>>> With less, it starts faster.
>>>>>
>>>>> If you do squid -k reconfigure and watch the proceess list, you will
>>>>> see that the squid dosen't work until all the childrens are
running...
>>>>> it would be great that the squids starts as soon as ONE child per
type
>>>>> is running... or to have a minimum childrens, and a maximum
>>>>> childrens...
>>>>>
>>>>> Sorry for my english, never study :)
>>>>>
>>>>> Luis Daniel Lucio Quiroz <luis.daniel.lucio_at_gmail.com> escribió:
>>>>>
>>>>> > Le mardi 3 novembre 2009 22:50:58, Amos Jeffries a écrit :
>>>>> >> Luis Daniel Lucio Quiroz wrote:
>>>>> >> > HI squids,
>>>>> >> >
>>>>> >> > We have 2 squid server, one with load other with minimal (1-2
>>>>> users).
>>>>> >> > After doing a -k reconfigure, the loaded server delays 40
>>>>> seconds, but
>>>>> >> > unloaded 2 seconds. Look:
>>>>> >> >
>>>>> >> > Unloaded:
>>>>> >> > 2009/11/03 19:01:14| Processing Configuration File:
>>>>> /etc/squid/squid.conf
>>>>> >> > (depth 0)
>>>>> >> > 2009/11/03 19:01:14| Processing Configuration File:
>>>>> /etc/squid/squid.acl
>>>>> >> > (depth 1)
>>>>> >> > 2009/11/03 19:01:14| WARNING: HTTP requires the use of Via
>>>>> >> > 2009/11/03 19:01:14| Initializing https proxy context
>>>>> >> > 2009/11/03 19:01:14| Store logging disabled
>>>>> >> > 2009/11/03 19:01:14| User-Agent logging is disabled.
>>>>> >> > 2009/11/03 19:01:14| Referer logging is disabled.
>>>>> >> > 2009/11/03 19:01:14| DNS Socket created at 0.0.0.0, port 49328,
>>>>> FD 8
>>>>> >> > 2009/11/03 19:01:14| Adding nameserver 127.0.0.1 from
>>>>> /etc/resolv.conf
>>>>> >> > 2009/11/03 19:01:14| Adding domain sat.gob.mx from
>>>>> /etc/resolv.conf
>>>>> >> > 2009/11/03 19:01:14| Adding domain insys-corp.com.mx from
>>>>> >> > /etc/resolv.conf 2009/11/03 19:01:14| helperOpenServers:
>>>>> Starting 32/32
>>>>> >> > 'squidGuard' processes 2009/11/03 19:01:15| helperOpenServers:
>>>>> Starting
>>>>> >> > 16/16 'digest_ldap_auth' processes
>>>>> >> > 2009/11/03 19:01:15| helperOpenServers: Starting 32/32
>>>>> 'squid_ldap_group'
>>>>> >> > processes
>>>>> >> > 2009/11/03 19:01:15| helperOpenServers: Starting 32/32
>>>>> >> > 'ip-user-helper.pl' processes
>>>>> >>
>>>>> >> Wow! thats a LOT of helper processes for 2 users.
>>>>> >>
>>>>> >> They start fast enough though.
>>>>> >>
>>>>> >> > 2009/11/03 19:01:16| Accepting HTTP connections at 0.0.0.0,
>>>>> port 3128,
>>>>> >> > FD 57. 2009/11/03 19:01:16| Accepting ICP messages at 0.0.0.0,
>>>>> port 3130,
>>>>> >> > FD 122. 2009/11/03 19:01:16| Accepting HTCP messages on port
>>>>> 4827, FD
>>>>> >> > 123. 2009/11/03 19:01:16| Accepting SNMP messages on port 3401,
>>>>> FD 124.
>>>>> >> >
>>>>> >> > Loaded:
>>>>> >> > 2009/11/03 18:59:05| Processing Configuration File:
>>>>> /etc/squid/squid.conf
>>>>> >> > (depth 0)
>>>>> >> > 2009/11/03 18:59:05| Processing Configuration File:
>>>>> /etc/squid/squid.acl
>>>>> >> > (depth 1)
>>>>> >> > 2009/11/03 18:59:05| WARNING: HTTP requires the use of Via
>>>>> >> > 2009/11/03 18:59:05| Initializing https proxy context
>>>>> >> > 2009/11/03 18:59:05| Store logging disabled
>>>>> >> > 2009/11/03 18:59:05| User-Agent logging is disabled.
>>>>> >> > 2009/11/03 18:59:05| Referer logging is disabled.
>>>>> >> > 2009/11/03 18:59:05| DNS Socket created at 0.0.0.0, port 35507,
>>>>> FD 8
>>>>> >> > 2009/11/03 18:59:05| Warning: Could not find any nameservers.
>>>>> Trying to
>>>>> >> > use localhost
>>>>> >> > 2009/11/03 18:59:05| Please check your /etc/resolv.conf file
>>>>> >> > 2009/11/03 18:59:05| or use the 'dns_nameservers' option in
>>>>> squid.conf.
>>>>> >>
>>>>> >> Oops! major problem with DNS on this server.
>>>>> >>
>>>>> >> > 2009/11/03 18:59:05| helperOpenServers: Starting 32/32
>>>>> 'squidGuard'
>>>>> >> > processes 2009/11/03 18:59:17| helperOpenServers: Starting
16/16
>>>>> >> > 'digest_ldap_auth' processes
>>>>> >> > 2009/11/03 18:59:23| helperOpenServers: Starting 32/32
>>>>> 'squid_ldap_group'
>>>>> >> > processes
>>>>> >> > 2009/11/03 18:59:34| helperOpenServers: Starting 32/32
>>>>> >> > 'ip-user-helper.pl' processes
>>>>> >> > 2009/11/03 18:59:47| Accepting HTTP connections at 0.0.0.0,
>>>>> port 3128,
>>>>> >> > FD 57. 2009/11/03 18:59:47| Accepting ICP messages at 0.0.0.0,
>>>>> port 3130,
>>>>> >> > FD 122. 2009/11/03 18:59:47| Accepting HTCP messages on port
>>>>> 4827, FD
>>>>> >> > 123. 2009/11/03 18:59:47| Accepting SNMP messages on port 3401,
>>>>> FD 124.
>>>>> >> > 2009/11/03 18:59:47| Pinger socket opened on FD 126
>>>>> >> > 2009/11/03 18:59:47| Configuring Parent 127.0.0.1/8080/7
>>>>> >> > 2009/11/03 18:59:47| Configuring Parent 10.10.50.232/8080/7
>>>>> >> >
>>>>> >> > I wonder to know if there is a way i can speed this up.
>>>>> >>
>>>>> >> An old rule-of-thumb in computing is to start with the first
>>>>> reported
>>>>> >> problem and see how many of the following disappearr
>>>>> >>
>>>>> >> For you that is as Squid suggests "Please check your
>>>>> /etc/resolv.conf
>>>>> >> file".
>>>>> >>
>>>>> >> The slow server seems to have no DNS servers available. This
>>>>> could be
>>>>> >> causing any kind of problems for the helpers later on. All of the
>>>>> extra
>>>>> >> delay is during the startup process of the helpers.
>>>>> >>
>>>>> >> Amos
>>>>> >>
>>>>> >
>>>>> > You were right, I was missing DNS, whoever, look, delay is still
>>>>> slow. about
>>>>> > 54 sec to reconfigure. What other think I may change. This is a
>>>>> loaded
>>>>> > server, with about 1k connections.
>>>>> >
>>>>> > 2009/11/04 11:15:35| Reconfiguring Squid Cache (version
>>>>> 3.0.STABLE19)...
>>>>> > 2009/11/04 11:15:35| FD 76 Closing HTTP connection
>>>>> > 2009/11/04 11:15:35| FD 141 Closing ICP connection
>>>>> > 2009/11/04 11:15:35| FD 142 Closing HTCP socket
>>>>> > 2009/11/04 11:15:35| FD 143 Closing SNMP socket
>>>>> > 2009/11/04 11:15:35| Processing Configuration File:
>>>>> /etc/squid/squid.conf
>>>>> > (depth 0)
>>>>> > 2009/11/04 11:15:35| Processing Configuration File:
>>>>> > /etc/squid/squid.acl (depth
>>>>> > 1)
>>>>> > 2009/11/04 11:15:35| WARNING: HTTP requires the use of Via
>>>>> > 2009/11/04 11:15:35| Initializing https proxy context
>>>>> > 2009/11/04 11:15:35| Store logging disabled
>>>>> > 2009/11/04 11:15:35| User-Agent logging is disabled.
>>>>> > 2009/11/04 11:15:35| Referer logging is disabled.
>>>>> > 2009/11/04 11:15:35| DNS Socket created at 0.0.0.0, port 60779, FD
8
>>>>> > 2009/11/04 11:15:35| Adding nameserver 127.0.0.1 from squid.conf
>>>>> > 2009/11/04 11:15:35| helperOpenServers: Starting 32/32
>>>>> 'squidGuard' processes
>>>>> > 2009/11/04 11:15:51| helperOpenServers: Starting 16/16
>>>>> 'digest_ldap_auth'
>>>>> > processes
>>>>> > 2009/11/04 11:15:59| helperOpenServers: Starting 32/32
>>>>> 'squid_ldap_group'
>>>>> > processes
>>>>> > 2009/11/04 11:16:15| helperOpenServers: Starting 32/32
>>>>> 'ip-user-helper.pl'
>>>>> > processes
>>>>> > 2009/11/04 11:16:31| Accepting HTTP connections at 0.0.0.0, port
>>>>> > 3128, FD 78.
>>>>> > 2009/11/04 11:16:31| Accepting ICP messages at 0.0.0.0, port 3130,
>>>>> FD 143.
>>>>> > 2009/11/04 11:16:31| Accepting HTCP messages on port 4827, FD 144.
>>>>> > 2009/11/04 11:16:31| Accepting SNMP messages on port 3401, FD 391.
>>>>> >
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> ---------------------------------------
>>>> Malte Schröder
>>>> MalteSch_at_gmx.de
>>>> ICQ# 68121508
>>>> ---------------------------------------
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> Jorge Armando Medina
>> Computación Gráfica de México
>> Web: http://www.e-compugraf.com
>> Tel: 55 51 40 72, Ext: 124
>> Email: jmedina_at_e-compugraf.com
>> GPG Key: 1024D/28E40632 2007-07-26
>> GPG Fingerprint: 59E2 0C7C F128 B550 B3A6 D3AF C574 8422 28E4 0632
>>
>>
>>
Received on Thu Nov 05 2009 - 02:06:09 MST