Re: [squid-users] cache of 404 and redirects

Adrian Chadd wrote:
> On Thu, May 08, 2008, Amos Jeffries wrote:
>>> On Wed, May 07, 2008, Christian Seifert wrote:
>>>> Or if you could merge 2.7 into 3.0, that would also help. Seems like 3.0
>>>> currently doesnt have the storeurl_program options....
>>> When the 3.0 developers push out a stable tree which people can move
>>> to, sure, I'll look at merging stuff of mine into -3.
>>>
>>> But right now I don't find -3 "fun" to work on,
>> There you have the crux of the entire (5 year?) Squid-2 vs Squid-3 debate.
>> Adrian doesn't find C++ code 'fun'.
>
> Hey cool! Somehow it became about C++.
>
> No, I don't find the development direction, the lack of meaningful pre-planning
> except when commercial interests are involved, the lack of useful large-scale
> user deployment feedback (and the lack of community participation in general is
> another topic, but I digress) and the fact that the codebase combines the worst
> bits of C and C++ in a monolithic, semi-structured mess "fun".

Lack of pre-planning? well that was fixed months ago. I don't know how
you missed it.
  http://wiki.squid-cache.org/RoadMap/Squid3

We have only a few having time for 'fun' in 3.x. The time I want to have
'fun' has mostly been lost when I cross-port your bits of 'fun' from 2.x
because people (one of your clients perhapse?) said it was also needed
in 3.x and you don't do 3.x.

>
> I find it "fun" that companies like Yahoo, last.fm and Wikipedia still run Squid-2
> in large scale production, and that the improvements I make in whatever I choose
> to work on in my spare time are involved in the day-to-day experiences of millions
> of people, even though none of them know about it.
>
> I find it "fun" that I can take a legacy bit of software and start to modernise
> it, far past what current user expectations are. I know I'm reinventing the wheel
> somewhat, but I find trying to reinvent the wheel whilst not reinventing the wheel
> a "fun" challenge.

Ah, thanks. Finally I find out what you mean by 'fun'.

>
> I started Xenion to try and turn some of that "fun" into something worthwhile,
> so I could spend more of my time working on things which may make a bit of
> a difference.

Um, just so you know. While I may be frustrated about the whole debate
and on the other side of the fence now when it comes to bug-fixing. What
I offered for Xenion from Treehouse is still on the table.

>
>>> and commercial work
>>> aside, I still hack on Squid for fun. Commercial work not aside,
>>> I don't agree with the directions that -3 is currently taking, and
>>> until there's a more sensible and stable codebase I can't justify
>>> suggesting that {current, new} clients migrate to Squid-3.
>> Come on, define 'sensible code' and stability for the readers please.
>> 2.7 is less stable than 3.0 at present. The old roadmap from 2.7 to 2.8
>> included sweeping core changes more destabilising than the 3.0->3.1 ones.
>
> Weird! 2.7 is stable enough, and faster, and the few bugs introduced are being
> ironed out.

Well, 'stable enough' covers every STABLE ever released. 'faster' there
is the one bonus amongst the chatter. 'few bugs' seems a lot like
3.0PRE6 and 3.0RC1 'just a few bugs'.

> Just like in 3.0. Except, 3.0 -> 3.1 is a large jump,

Aha, dropping 150K lines of code because its no longer needed was a big
change. At least the 45k of new code that has gone in is doing so with
up to 12 months of testing behind it. Production I might add, because
the largest parts of that new code have by commercial and legal
requirement been in use for at least 4 months in more than a handful of
sites around the world.
The recent spat of 3-HEAD bugs are the third wave of testers to migrate
to 'experimental' 3.1 code.

> and 3.1 -> 3.2
> is going to be an even larger jump.

No argument there from me. Most of that will be file renaming and code
shuffling (no changes!). To make those 'distinct API' in 3.x obvious to
everybody.

> The last couple of 2.7 bugs have been due to
> recent additions on contract; and Henrik's found and squished those.
> I personally think 2.7.STABLE1 should've been released a couple of months ago.
> As it stands right now, I think 2.7 is going to match and exceed stability and
> performance expectations from Squid-2.6, and thats what I find "fun".
>
> Study 2.7 to 2.HEAD; the changes aren't all that drastic. The biggest drastic
> change is the elimination of the store -> client copy, which yes there are bugs,
> but they're being worked out.

The 3.1 ChangeLog is similarly very short (for 16 months work). Two
major changes, and a handful of smaller features involved with bringing
the Protocols squid handles up to coping with modern web traffic. Many
of which required the major changes.
We've just done the features over a longer incremental period, with more
testing than before.

>
> Insted of trying to squeeze more into Squid-2.HEAD (and Cacheboy, for that
> matter), I've put a hold on any of my work that'll destabilise things further.
> I spent time working in a sandbox to see exactly how far I could get with
> sweeping changes; I took notes, I learnt from the experience, and the stuff
> thats different between 2.HEAD and Cacheboy right now represents what I've
> learnt from that experience.
>
> Check out the Cacheboy stuff - that includes a lot of the changes which I
> was going to make along the Squid-2.8 roadmap, and almost none of the changes
> were dramatic. Code reorganisation, code tidyup, replace a FIFO implementation
> with something slightly more sensible. Break out the code in preparation for
> further development and testing work. All the stuff that we should've done years
> ago, except the focus shifted to -3 and, and I'm going to really honest here,
> it never delivered.
>
> Sensible code: modular, well-defined APIs, try to remove legacy stuff whilst
> maintaining stability. Squid-2 and Squid-3 haven't tackled the big issues
> in the codebase, and it doesn't seem like Squid-3 will tackle any of those
> issues any time soon.

Have you audited any of the code in my 3.1 testing changes, or upcoming
cleanup branch. Or Alex's lib'ing branch? I'm sure you have at least
seen the planning discussions going by for months now. Thats the 3.2
stability + better API's issue right there.

The one major lesson I have learned while developing the testing of 3.1
modules is that the parts which have been converted DO have well-defined
modular API's. It's just the files are all largely stuffed into src/ ,
undocumented, and that is being fixed in 3.2.

>
> So no Amos, I don't buy the "2.7 to 2.HEAD is just as destabilising as 3.0
> to 3.1" line.

I did not say that. I said 'the planned road map changes'. We have no
exact measure for them yet. But judging from a few facts:

   - a number are cross-ports of stuff from 3.x (ironically: the bits
you hold out as paradigms of instability during the last argument)

   - the remainder are largely about changing parts of squid which are
highly similar for 2.x and 3.x. (ironically: they have been left out of
3.x so far because of the high degree of instability that changing it
would cause.)

Based on those, I'll keep my opinion thanks. At least until we are able
to judge on some numbers.

> I believe that anyone who bothers reading the code/changesets will
> come to the same conclusion.

It depends on what has been touched, and how many bugs come out. Which
we will nt have a measure for until after the fact. I just did a quick
release comparison of what appears serious bugs and 2.7 had a count
roughly equal to 3.1 so far. 2.6 was higher than 3.0, but probably
because of the time its been out.

For the numbers we can tell. Since each branched out of 2.5:
  2.x had 35123 line added.
  3.x had 54365 lines removed.

Each incremental release of 2.x has had roughly 100k lines of code
changed. Subtracting all the new RFC from 3.x it has roughly the same.

I did bother to read the changesets, and came to the conclusion that 3.0
was more stable. Simply counting the mentioned changes for 4 months
since STABLE1 came out shows:
  2.6 had 249
  3.0 had 104

The number of bugs are similar:
  2.6 had 175
  3.0 had 57

(counting the period of 2.6 selected to match the period of information
available to 3.0. Namely the first 10 weeks since STABLE 1)

Comparing 2.7 as a development version is a littel unfair, but if you
like its counts are already nearly twice the 2.6 counts.

On the lighter side. Looking at the changeset titles, early 3.0 seems to
be mostly Windows and portability. While early 2.6 seems to be mostly
segfaults and memory leaks.

I don't really know what opinion you expect people to get from browsing
those really. Other than 3.0 has less major problems.

>
>> >From the stats 12% of the squid users (on Linux with basic packages) are
>> using 3.0 now over all versions of 2.x. _On_top_ of the 10% increase in
>> total users 2.x (all versions) had over the period since 3.0 was released.
>
> Origin please. I bet you can attribute a large part of that to people who
> run the "latest" version of software, and Squid-3 probably runs fine for
> them.

http://qa.debian.org/popcon.php?package=squid
http://qa.debian.org/popcon.php?package=squid3

Based on raw total installs of squid-common (core files package):
    ( (3.0 installed - 3.0 old ) / (2.6 installed - 2.6 old) )*100 =
    (574/4692) *100 = 12.2%

'recent' numbers are people who run the 'latest'. Looks like a low 3.8%
to 5% of each versions users.
'installed' is total current installs, 'old' is people who install but
don't use for some reason.

and for the graphs:
http://people.debian.org/~igloo/popcon-graphs/index.php?packages=squid&show_installed=on&show_old=on&show_recent=on&want_percent=on&want_legend=on

http://people.debian.org/~igloo/popcon-graphs/index.php?packages=squid3&show_installed=on&show_old=on&show_recent=on&want_percent=on&want_legend=on

Though note the precentages it refers to directly are percentage of
total measured server market. Including dedicated mailservers, web
servers, db servers etc.

>
>>> (This is why I forked off Squid-2 into Cacheboy, btw.)
>> And why we concreted, tested, and fixed up squid-3 to released 3.0 full of
>> the existing stable changes. Prior to adding much of the fancy new stuff
>> from 2.6, 2.7, 2.8, and 2.9.
>
> The stuff was added to 2.x because people wanted/coded it. You know, because
> people are actively using it and stuff.

Don't forget the ... and were given no incentive to port it to 3.x or to
develop it there directly.

Well, that's the commercial side. Nobody has argued against sponsored
additions to 2.x. Just to playing around with completely new experiments
in the 2.x codebase.

>
>> We in squid-3 agree with you Christian. If only the 2.6 and 2.7 changes
>> were also ported to 3.x when they had passed testing in 2.x there would be
>> not quite such a mess of shifting goalposts and last-minute catchup
>> features plugged on top of (de-stabilizing!) 3.1.
>
> Hey, I'm really sorry that the stuff _I_ did for fun in a _stable_ codebase
> weren't done in an _unstable_ codebase which I then couldn't really suggest
> that people try out in production. I'm also sorry for those feature submissions
> for Squid-2.6 and Squid-2.7 - obviously thats not what people want to use.

They want to use what they are told by an expert is the best for them.
Of what they think is the best anyway. All we have to do is make sure we
speak about all the options. Not just one.
Solicitation will always gain clients for what is solicited. Regardless
of alternatives being available. You know this. It's what keeps spam alive.

You are the only person who keeps referring to 3.x as 'unstable' still.
Knowing full well that large parts (the most 'messy' bits at that!) are
almost identical to 2.x . Differing only in some bug fixes applied, and
features added in top to 2.6.

The changed bits in 3.0 are stable enough for production use. The draw
for 2.6 on production-environment grounds is solely speed performances
you added to 2.x since 3.0 closed for stabilization.

>
> Its the Squid-3 developers' choice to throw stuff into 3.1 to try and become
> feature-par with Squid-2. Hey, I'm all for that, but there's so much crap
> in the Squid-ALL codebases which need tidying up before the codebase can really
> shine, and I'm quite frankly fed up waiting for the right oppertunity to do it.

So far I have not been around for the whole history of it, but it seems
to me that every time the opportunity comes up so do all these arguments
against taking it. Ending full-circle with 'I don't find -3 "fun" to
work on'.
The door is always open. The only ask is that new experimental stuff
gets tested on 3-HEAD.

>
> Its crap like this which makes me simply want to quit developing Squid again.
>
> Adrian
>

I guess what I'm trying to say is that the actual change in 3.x is
nowhere near what its hyped up to be.

Amos

-- 
Please use Squid 2.6.STABLE20 or 3.0.STABLE5