Re: Memory problems

Hello

I checked this with Duane, so hopefully this is correct ;)

This will hopefully provide some info as to the memory problems that people
have been upset about...

Duane Said:
=======================================================================
Yes, this is basically correct. The pools are really just to
try avoiding some malloc/free overhead and fragmentation because
there are certain block sizes which we allocate frequently.

The size of the pools are choosen from the 'cache_mem' and SQUID_MAXFD
values. If a pool has more blocks allocated than its limit, it calls
free(), otherwise it pushes the unused block onto a stack.

Another problem is that in-transit objects don't use the full 4k blocks
allocated to hold them. This is why memory usage used to be so
unpredictable. We would count the number of bytes for in-transit
objects, and sort of ignore the unused partial sections of the 4k
blocks.

=========================================================================
One of the messages on the subject said:

> > >>2) Why is it not releasing unneeded memory back to the OS when the
> > >>memory_pools option is set to 'off'
> > >
> > >this I can answer: Very few malloc() implementations actually
> > >release memory back to the OS when you free(), until the process
> > >exits.
>
> If i understand correctly.
> Squid grows only if it uses more memory then used before. It keeps the
> maximum size it reached once before. I cant believe this really as then
> the size of the squid would once grow VERY fast in some minutes and
> sometimes takes days. But my squids keep growing very slowly about a
> week then they are min. 20MB (At start 4.5MB). If there would be a
> bug in malloc implementation why dont i see this with other processes
> like named, apache, cron etc ?

=========================================================================
My reply to the message (hopefully the interesting bit ;)

In this example I am going to tell squid (from my config file) to use 16
megs of Ram.

Squid has a "block size" of the smallest amount of Ram it will allocate at
once. (this stops fragmentation etc). When a part of squid needs more ram,
it doesn't call "malloc", but calls a function that will allocate it a
farily large "block" of ram. When the function no longer needs this ram,
it returns the whole block. This will stop malloc "losing" memory as
millions of tiny blocks are "malloc"d and "free"d. When a function "gives
the memory back" to the memory management function, we don't
"free" it (is this where the "free memory pools" stuff comes in?), but
put it back into our list of blocks that we can allocate when next a function
requests ram.

Now - say someone downloads a 100 meg file through the cache.

Our function that does the download will allocate ram up to it's limit of
16 megs, and our function realises that it has no ram in it's "pool" left.
When this happens, it allocates another fairly large block, and then gives
some of that to the download function. This will happen a fair number of times
,
and eventually it will have 100+ megs of memory that has been allocated to
squid, and on it's internal table, it will see that it has 100 megs of
blocks allocated to "function x", and they cannot be used. When "function x"
finishes the download, it tells the memory management function that it
can now use those blocks again, but the memory function doesn't call "free",
and reduce squid's size the "ps" table, it just marks them as available
nect time it wants to do a download. (ie, you could download 2 100 meg files
directly after another, and the size wouldn't increase substantially on the
second download). These "free" blocks are then available for use.

The above explains sudden jumps in memory usage.

The problem is that not ALL functions use the memory management function to
get memory, and often call "malloc" and "free" repetitively, so you do get
cases where squid grows slowly but substantially. This is the problem with
older versions of FreeBSD and AIX, where after a week or so you realise that
it is using 400 megs of ram, and can only account for 30 of it...
These are the cases where you need to use another malloc.

You can tell the difference by looking at "VM Objects" list on your cache
manager during peak times. If you look at the "size" value on each object,
you will see how much ram currently downloaded files are taking. You quite
often find that there are people downloading huge files (for eg, quake),
and this is taking 10 megs of ram.

        Oskar

=============================================
'experience made art, but inexperience luck.'
=============================================
Received on Mon Dec 02 1996 - 00:58:22 MST