Re: [squid-users] oddity with squid 3.3.2 and https

On Tue, Apr 16, 2013 at 03:16:47PM +0930, Brett Lymn wrote:
> On Tue, Apr 16, 2013 at 04:16:49PM +1200, Amos Jeffries wrote:
> > >
> > >The user_rewrite.pl script just strips off the ntlm/kerberos bits in
> > >front of the username so the upstream "security" device sees a username
> > >it understands. I put the ACL on the never_direct because it is a
> > >"slow" ACL evaluation and fires the rewrite in a, supposedly, harmless
> > >manner.
> >
> > >Not sure why only https is affected, everything did work with squid
> > >3.1.19.
> >
> > Probably not just HTTPS. The warning will occur on any request being
> > passed to this ACL without authentication credentials.
>
> Well, for some reason only peer selection for unauthenticated CONNECTs
> fails. I am not sure why, I have no rules for cache peer access that
> involve CONNECT.
>

I have done some more debug with this and can rule out the
authentication rewrite playing a part but I have found something very
puzzling around peer selection for CONNECT (and, possibly, POST and
FTP).

Firstly, our proxy hierarchy goes something like this:

1) two parent proxies in the DMZ that handle external WWW access
2) two parent proxies in the UK that handle some intranet access
3) four internal security gateways to scan/block access, thes parent off
   the DMZ parents
4) four internal squid proxies that handle all the clients and route
   traffic to the uk for intranet, to the security gateways for most
   traffic and direct to the DMZ parents for traffic we don't want going
   through the security gateway (bypass).

To do this I have a bunch of text files referenced by dstdomain or dst
acls (some access is via bare IP). I have the peers set up like so:

cache_peer sec-gw1 parent 8080 7 login=*:password no-query userhash
cache_peer sec-gw2 parent 8080 7 login=*:password no-query userhash
cache_peer sec-gw3 parent 8080 7 login=*:password no-query userhash
cache_peer sec-gw4 parent 8080 7 login=*:password no-query userhash
cache_peer dmz-parent1 parent 3128 0 round-robin default
cache_peer dmz-parent2 parent 3128 0 round-robin
cache_peer uk-parent1 parent 3128 0 no-query
cache_peer uk-parent1 parent 3128 0 no-query

I then do the routing using cache_peer_access, like so:

cache_peer_access uk-parent1 allow uk_intranet
cache_peer_access uk-parent1 allow uk_dst
cache_peer_access uk-parent1 deny all
#
cache_peer_access uk-parent2 allow uk_intranet
cache_peer_access uk-parent2 allow uk_dst
cache_peer_access uk-parent2 deny all
#
cache_peer_access dmz-parent1 allow Bypass
cache_peer_access dmz-parent1 allow BypassIP
cache_peer_access dmz-parent1 deny all
#
cache_peer_access dmz-parent2 allow Bypass
cache_peer_access dmz-parent2 allow BypassIP
cache_peer_access dmz-parent2 deny all
#
cache_peer_access sec-gw1 deny uk_intranet
cache_peer_access sec-gw1 deny uk_dst
cache_peer_access sec-gw1 deny Bypass
cache_peer_access sec-gw1 deny BypassIP
cache_peer_access sec-gw1 allow all
#
cache_peer_access sec-gw2 deny uk_intranet
cache_peer_access sec-gw2 deny uk_dst
cache_peer_access sec-gw2 deny Bypass
cache_peer_access sec-gw2 deny BypassIP
cache_peer_access sec-gw2 allow all
#
cache_peer_access sec-gw3 deny uk_intranet
cache_peer_access sec-gw3 deny uk_dst
cache_peer_access sec-gw3 deny Bypass
cache_peer_access sec-gw3 deny BypassIP
cache_peer_access sec-gw3 allow all
#
cache_peer_access sec-gw4 deny uk_intranet
cache_peer_access sec-gw4 deny uk_dst
cache_peer_access sec-gw4 deny Bypass
cache_peer_access sec-gw4 deny BypassIP
cache_peer_access sec-gw4 allow all

(hopefully I have made this consistent - I have modified the names here
to protect the not so innocent)

We have a bunch of sites that are allowed access without requiring
authentication. What I have been finding is that CONNECT without
authentication connections fail, enabling debug 44 in the squid.conf
I see:

2013/04/24 09:56:33.498 kid1| peer_select.cc(156) peerSelect: peerSelect: CONNECT
2013/04/24 09:56:33.498 kid1| peer_select.cc(442) peerSelectFoo: peerSelectFoo: 'CONNECT 12.x.x.x'
2013/04/24 09:56:33.498 kid1| peer_select.cc(447) peerSelectFoo: peerSelectFoo: direct = DIRECT_UNKNOWN (always_direct to be checked)
2013/04/24 09:56:33.498 kid1| peer_select.cc(209) peerCheckAlwaysDirectDone: peerCheckAlwaysDirectDone: DENIED
2013/04/24 09:56:33.498 kid1| peer_select.cc(442) peerSelectFoo: peerSelectFoo: 'CONNECT 12.x.x.x'
2013/04/24 09:56:33.498 kid1| peer_select.cc(453) peerSelectFoo: peerSelectFoo: direct = DIRECT_UNKNOWN (never_direct to be checked)
2013/04/24 09:56:33.498 kid1| peer_select.cc(186) peerCheckNeverDirectDone: peerCheckNeverDirectDone: ALLOWED
2013/04/24 09:56:33.498 kid1| peer_select.cc(192) peerCheckNeverDirectDone: direct = DIRECT_NO (never_direct allow)
2013/04/24 09:56:33.498 kid1| peer_select.cc(442) peerSelectFoo: peerSelectFoo: 'CONNECT 12.x.x.x'
2013/04/24 09:56:33.498 kid1| peer_select.cc(682) peerGetSomeParent: peerGetSomeParent: CONNECT 12.x.x.x
2013/04/24 09:56:33.498 kid1| Failed to select source for '[null_entry]'
2013/04/24 09:56:33.498 kid1| always_direct = DENIED
2013/04/24 09:56:33.498 kid1| never_direct = ALLOWED
2013/04/24 09:56:33.498 kid1| timedout = 0

And that is it - no parent selected but I believe that any of the
security gateway peers should have been selected in this case because
the address is not in the BypassIP nor in uk_dst. What is weird is if I
add the line:

cache_peer_access sec-gw1 allow CONNECT

before any other cache_peer_access lines for sec-gw1, i.e.:

cache_peer_access sec-gw1 allow CONNECT
cache_peer_access sec-gw1 deny uk_intranet
cache_peer_access sec-gw1 deny uk_dst
cache_peer_access sec-gw1 deny Bypass
cache_peer_access sec-gw1 deny BypassIP
cache_peer_access sec-gw1 allow all

then sec-gw1 will be selected as a peer when accessing the 12.x.x.x
address. What is even weirder is if I do this:

acl foo dstdomain foo.bar.com

cache_peer_access sec-gw1 deny foo
cache_peer_access sec-gw1 allow CONNECT
cache_peer_access sec-gw1 deny uk_intranet
cache_peer_access sec-gw1 deny uk_dst
cache_peer_access sec-gw1 deny Bypass
cache_peer_access sec-gw1 deny BypassIP
cache_peer_access sec-gw1 allow all

then sec-gw1 WILL NOT be selected as a peer if I access the 12.x.x.x
address - note the foo acl is literally what I put, there is no relation
to the 12.x.x.x address at all, it should not match.

As I said before, this seems to be something that has happened since
3.1.19. I know that CONNECT and POST are affected because I have had
problems with both. Mostly if I put an explicit never_direct for the
unauthenticated access then it seems to get the parent selection right
but not for everything.

-- 
Brett Lymn
"Warning:
The information contained in this email and any attached files is
confidential to BAE Systems Australia. If you are not the intended
recipient, any use, disclosure or copying of this email or any
attachments is expressly prohibited.  If you have received this email
in error, please notify us immediately. VIRUS: Every care has been
taken to ensure this email and its attachments are virus free,
however, any loss or damage incurred in using this email is not the
sender's responsibility.  It is your responsibility to ensure virus
checks are completed before installing any data sent in this email to
your computer."