Re: [squid-users] "Quadruple" memory usage with squid

Linda started this thread with huge performance problems
when Squid with a size of 12 GB forks 15 times.

Linda emailed me that she is doing a test with
vm.pmap.pg_ps_enabled set to 1 (the kernel will
transparently transform 4K pages into superpages)
which gives a big relief for TLB management
and will most likely reduce performance problems with
forks of very large processes.
I am very interested in the result of the test...
But independently if the transparent allocation of huge pages
will give the needed performance gain, forking 12 GB
is not nothing. Doing it 15 times is ... a lot of housekeeping.

Therefore I have just a silly question:
can we modify Squid to fork at a slower pace?
It would be interesting to see if a 0.2 second gap
between each fork gives the system a enough
time to make the fork of 12 GB happen before the next fork
is done. Perhaps with a little "intelligence":
no delay between forks with a cache_mem < 512 MB,
0.1 sec delay with cache_mem < 4 GB etc.

Any ideas ?

Marcus

Robert Collins wrote:
> On Wed, 2009-11-25 at 02:11 +0100, Henrik Nordstrom wrote:
>> ons 2009-11-25 klockan 09:07 +1100 skrev Robert Collins:
>>> On Tue, 2009-11-24 at 13:45 +0100, Henrik Nordstrom wrote:
>>>> tis 2009-11-24 klockan 15:06 +1100 skrev Robert Collins:
>>>>
>>>>> http://www.netbsd.org/docs/kernel/vfork.html has some interesting notes
>>>>> from the BSD world about this.
>>>> vfork is fundamentally broken.
>>> Beyond the obvious (that it doesn't separate the memory out?)
>> Undefined results if any of the following is used:
>>
>> - threads
>> - signals
>> - any form of output
>> - pretty much any other syscall than an successful execve
>
> In the 'vchild', yes. However I don't see why that would cause us
> problems: in the child side of the vfork we would only be calling
> execve / execvp - which will be fine.
>
>>>> there is other alternatives coming, getting around the virtual memory
>>>> issue when starting new processes.
>>> What are they called?
>> Searching.. posix_spawn() and it's posix_spawnp() wrapper seems to be
>> the one.
>
> They use vfork (POSIX_SPAWN_USEVFORK) or fork internally. vfork has been
> available for longer and should be trivial to drop in as an experiment,
> whereas posix_spawn is still not widely available (and has precisely the
> same issues for us).
>
> -Rob
Received on Wed Nov 25 2009 - 02:52:30 MST