Squid squid-1.NOVM.19 ate my cache/log

Perhaps I haven't been paying attention, but it seems to me 1.19 has
some serious lossage--in fact, it lost most of our cache disk. I just
upgraded a small cache from 1.NOVM.18 to 1.NOVM.19, and when I started
up the new version for the first time it logged

97/12/26 13:38:00| Starting Squid Cache version 1.NOVM.19 for mips-dec-ultrix4.4...
97/12/26 13:38:00| With 256 file descriptors available
[...]
97/12/26 13:38:02| Configuring Parent pb.cache.nlanr.net/3128/3130
97/12/26 13:38:04| 4096 Lines read so far.
97/12/26 13:38:05| 8192 Lines read so far.
97/12/26 13:38:06| 12288 Lines read so far.
97/12/26 13:38:06| Finished rebuilding storage from disk image.
97/12/26 13:38:06| 13239 Lines read from previous logfile.
97/12/26 13:38:06| 683 Objects loaded.
97/12/26 13:38:06| 0 Objects expired.
97/12/26 13:38:06| 0 Duplicate URLs purged.
97/12/26 13:38:06| 0 Swapfile clashes avoided.
97/12/26 13:38:06| Took 4 seconds ( 170.7 objects/sec).
97/12/26 13:38:06| store_swap_size = 6964k

so I had 13,000 objects in the old cache, and 1.NOVM.19 loaded 700 of
them, losing track of 95% of the cache.

The cause of this seems to be the change

        - Changed storeInit() to allocate 1.5 times the estimated number
          of cache objects for the 'filemap' bitmap.

which changed MAX_SWAP_FILE from a constant 1<<21 to
3/2*maxSize/avjObjectSize. Other effects of this change have been
discussed recently, and it does significantly reduce memory usage for
small caches (a noticeable win for us), but I do not remember seeing
any mention of how this interacts with

        if (sfileno < 0 || sfileno >= MAX_SWAP_FILE)
            continue;

in storeDoRebuildFromDisk(). The effect is that any files that were
given numbers beyond the newly caclulated MAX_SWAP_FILE are quietly
discarded from cache/log and the files forgotten about but *not*
deleted. Worse, since those file numbers are now outside the active
range, they will never be reused, so the forgotten files will just
sit there until someone intervenes.

If I've got this right, it seems to me there ought to be a big warning
in CAPITAL LETTERS attached to 1.19 that

 - upgrading may cause squid to lose track of big chunks of
   cache disk space
 - decreasing maxSize or increasing avjObjectSize may lose
   track of smaller chunks of disk space

Since I was down to 700 objects that squid still had track of, and the
old log had been overwritten, I ended doing an 'rm -rf cache' followed
by a 'squid -z' to get back to a well defined state.

-- 
Dan Riley                                         dsr@mail.lns.cornell.edu
Wilson Lab, Cornell University       HEPNET/SPAN: lns598::dsr (44630::dsr)
http://www.lns.cornell.edu/~dsr/ "Distance means nothing/To me" -Kate Bush