Re: Hmmmm.

In message <E0w6dL0-0005ws-00@chacal.noc.demon.net>, James R Grinter writes:
> On Mon 17 Mar, 1997, "Michael O'Reilly" <michael@metal.iinet.net.au> wrote:
> >accept()'s. Anyone looking at this? It'd be a lot more portable than
> >the async disk I/O I see some people looking at...
>
> Look at doing the same trick for I/O to files as per dnsserver.
> The commercial harvest uses that with a 'filepumpd' process that
> does the requests.
>
> Otherwise, you have hellish locking problems between your shared
> memory (SysV shm would be better than mmap)- and practically all
> OSs fail when they start using semaphores heavily...

Yes, kicking it around last night that was one idea we came up
with. Some other as food for discussion.

#1. Something that's dead easy to do that'll give an immediate 20%
improvement on our box is to has a seperate process doing the
unlink()'s. i.e. spawn off
        perl -nle unlink
and write the name of the files to the process instead of trying to do
it in squid. Nice and easy to do.

The crazy ones start here. :)

#2. For machines with decent VM. Fork off a few children that expire
after 5 mins. have all the children parent thru the parent
process. The child processes never write to disk.

This means that all cache hits come from the children, the parent is
only processing misses, and only writes to disk, and should never
read. Problems are if you have N children, and the parent writes to a
shared memory page, you'll get N copies.

This is only possible on machines that have a good COW VM system.

#3. A seperate process doing all the disk I/O. Send commands for
read/writes to it. Means that you'll never block for disk I/O in the
server.

Problem is that it'll be a bloody mess to write!

You could start off just handling reads that way. but either way,
messy.

#4. as mentioned above. mmap() to share memory. child processes all
parent via parent process, children never write to MMAP, only
read. Means that you should be able to get away without locking if the
parent ensures that it never updates in the wrong order. Children
never write objects to disk, only read cache hits, and proxy-only
requests.

This is only useful on machines getting a fairly high cache hit ratio,
or machines that have lots of things they proxy-only for.

The best and the hardest is the seperate daemon doing disk. The one
that can easily be done right now is unlink as a seperate process.

Michael, kicking ideas safe in the knowledge that he doesn't have time
to do any of them. :(
Received on Mon Mar 17 1997 - 23:36:54 MST