Re: [squid-users] Linux + TPROXY + Remote Squid

NP Daniel,
i'm happy that more people are getting into it.
i have done my research now and i have used the next:
fedora| 16 p2p1(eth0) 192.168.15.100 gw 192.168.15.1

router\fw debian squeeze| eth1 192.168.15.1
2.6.32-5-amd64 | eth2 192.168.16.1
                        | eth0 192.168.10.107 gw 192.168.10.201

same debian as proxy |eth0 192.168.16.100 gw 192.168.16.1
squid 3.2.0.17

the 192.168.10.201 gw is a Gentoo AIO machine connected to DSL
and masquerading.
AIO = (proxy dhcp dns storage testing)
the AIO has a static route to the 192.168.15+16.0/24 nets using the
192.168.10.107

stay with me because it's complicated.
i used iptables marks and custom routing tables
iptables -t mangle -N FROMINT
iptables -t mangle -A FROMINT --jump MARK --set-mark 3
iptables -t mangle -N FROMPXY
iptables -t mangle -A FROMPXY --jump MARK --set-mark 2
iptables -t mangle -N TOPXY
iptables -t mangle -A TOPXY --jump MARK --set-mark 1
iptables -t mangle -A PREROUTING -i eth1 -p tcp -m tcp --dport 80 --jump
TOPXY
iptables -t mangle -A PREROUTING -i eth2 -p tcp -m tcp --dport 80 --jump
FROMPXY
iptables -t mangle -A PREROUTING -i eth0 -p tcp -m tcp --sport 80 --jump
FROMINT

these rules are marking the packets so the routing level can identify them
mark 1 is from client on eth1 and destined to port 80\http
mark 2 is the "next hop" from the proxy destined to the web
mark 3 is the packets that coming from the web server back to the client
but needed to be routed into the proxy back.

then i assigned every mark a routing table to handle it.

i had to create the routing tables in /etc/iproute2/rt_tables
##start
100 topxy
120 frompxy
140 fromint
255 local
254 main
253 default
##end

ip rule add fwmark 1 table topxy
ip rule add fwmark 2 table frompxy
ip rule add fwmark 3 table fromint

now this is a very difficult part to understand

anything that comes and destined to the proxy must go to the proxy and
should not have any other routing options so the topxy table should have
only route to the proxy
so:

ip route add default via 192.168.16.100 dev eth2 table topxy

but if anything comes from the proxy it should be sent to destination so
we copy the original routes using:

ip route show table main | while read ROUTE ; do ip route add table
frompxy $ROUTE; done

i could have used the same table for topxy and fromint because it has
the same rule that redirects the traffic to the proxy but i like it
organized so:

ip route add default via 192.168.16.100 dev eth2 table fromint

this is very good for a real routing environment but if you start nating
it will get messy. why ???
because a packet that was in the kernel before will not be
natted\masqueraded on the output interface.

i dont know if the problem is in the kernel\iptables\routing or what.

so there is a problem with natting.

i'm still wondering on th ipv6 thing
has anyone tried to intercept ipv6 traffic using tproxy??

Eliezer

On 01/06/2012 01:10, Daniel Niasoff wrote:
> Hi All,
>
> I'm jumping in the middle here but I have tproxy working with a separate router as follows;
>
> Here are the rules from my router.
>
> # Don.t mark webcache traffic
> $IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp --dport 80 -s $SQUID
> # Internal subnets to exclude
> $IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp --dport 80 -d 192.168.0.0/16 #Don.t cache internal
> $IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp --dport 80 -s 192.168.0.0/16 #Don.t cache internal
>
> # Now mark our traffic
> $IPTABLES -t mangle -A PREROUTING -j MARK --set-mark 5 -p tcp --dport 80
>
> ip rule add fwmark 5 table 5
> ip route add default via $SQUID dev bond0.8 table 5
>
> On my squid box I have the following Iptables rules
>
> *mangle
> :PREROUTING ACCEPT [47356:2123379]
> :INPUT ACCEPT [44233:3551720]
> :FORWARD ACCEPT [14057:711976]
> :OUTPUT ACCEPT [27932:3507208]
> :POSTROUTING ACCEPT [42005:4222687]
> :DIVERT - [0:0]
> -A PREROUTING -p tcp -m socket -j DIVERT
> -A PREROUTING -p tcp -m tcp --dport 80 -j TPROXY --on-port 3129 --on-ip 0.0.0.0 --tproxy-mark 0x1/0x1
> -A DIVERT -j MARK --set-xmark 0x1/0xffffffff
> -A DIVERT -j ACCEPT
> COMMIT
>
> #This makes sure that the traffic comes back to this squid box.
> *nat
> :PREROUTING ACCEPT [627:29853]
> :POSTROUTING ACCEPT [46:2562]
> :OUTPUT ACCEPT [93:5582]
> -A POSTROUTING -s 10.0.0.0/8 -o eth0 -p tcp -m tcp --dport 80 -j SNAT --to-source #SQUIDIP
> COMMIT
>
> ip -f inet rule add fwmark 1 lookup 100
> ip -f inet route add local default dev eth0 table 100
>
> This system works well for me and I have multiple squid proxies is a transparent load balanced config (using Linux virtual server)
>
> I've had 10 or 15 users testing it and with no complaints so far but I had to use the latest source code (not build 3.2.0.17)
>
> Hatzlacha
>
> Daniel
>
>
> -----Original Message-----
> From: Eliezer Croitoru [mailto:eliezer_at_ngtech.co.il]
> Sent: 31 May 2012 21:17
> To: Thomas York
> Cc: squid-users_at_squid-cache.org
> Subject: Re: [squid-users] Linux + TPROXY + Remote Squid
>
> the marking is not all the thing.
> you should also use routing tables based on the marking so in the prerouting mangle you mark and then the routing tables are compatible with the routing table.
>
> i will think of something.
> but it's out of the scope of squid and moving to routing.
>
> Eliezer
>
> On 31/05/2012 19:26, Thomas York wrote:
>> A TPROXY isn't useless just because I'm using NAT. The whole point of
>> using TPROXY is that it will also work with IPv6 (since iptables lacks
>> NAT capability with IPv6, which is fine). I'm marking and diverting
>> connections from eth2, because that's the interface that has clients connected to it.
>>
>> I had a separate table and marking for return port 80 data on the
>> router (eth0), but it didn't make any difference. I had the same issue
>> that I have now. As of this point, the only working solution I've seen
>> is to consolidate the proxy and the router. I REALLY hate to do this,
>> but at this point it looks like my only working solution.
>>
>> -- Thomas York
>> -----Original Message-----
>> From: Eliezer Croitoru [mailto:eliezer_at_ngtech.co.il]
>> Sent: Thursday, May 31, 2012 12:03 PM
>> To: Thomas York
>> Cc: squid-users_at_squid-cache.org
>> Subject: Re: [squid-users] Linux + TPROXY + Remote Squid
>>
>> well as i was suspecting you made the scenario from a movie.
>> in the real world you will design it a little different.
>> here a picture on the net:
>> http://cloud.ngtech.co.il/public.php?service=files&token=d88ff9e412a47
>> a2842c b8ac7137c6227f196d8f2&path=/squid-net.png
>>
>>
>> in yout specific case you are natting the internet access anyway so a
>> tproxy is useless.
>> i'm still trying to understand why you are marking and diverting the
>> connections from eth2.
>>
>> you have one problem in this specific case.
>> you are trying to do some marking that will direct the clients into
>> this router in a loop from the clients to router and then from router
>> to squid and from squid to box using the same source ip of the client
>> in this part you will get a big loop.
>>
>> what you should do is a marking of packets coming from each interface
>> differently and by this mark define 2 routing tables:
>> one that marked for squidbox because it came from the eth1 1 another
>> mark is for packets that are comming on interface eth5 and are to port
>> 80 will be marked 2 and will be routed\nated to the gw on eth0 another
>> mark is on the
>> eth0 interface when packets are coming from the http server it should
>> be routed to squidbox.
>>
>> if you do ask me i would have put squidbox in the eth0 net and do the
>> nat on squidbox instead of on the router.
>>
>>
>> Eliezer
>>
>>
>> text summary:
>> win7 eth0 10.1.1.253/24
>> gw 10.1.1.254
>>
>> FW
>> eth0 10.1.17.158/24
>> gw 10.1.17.254
>>
>> eth1 10.1.1.254/24
>> eth5 10.0.1.254/24
>>
>> iptables:
>> *nat
>> -A POSTROUTING -o eth0 -j MASQUERADE
>> COMMIT
>>
>> *mangle
>> :PREROUTING ACCEPT [126:15633]
>> :INPUT ACCEPT [126:15633]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [67:8420]
>> :POSTROUTING ACCEPT [67:8420]
>> :DIVERT - [0:0]
>> -A PREROUTING -i eth2 -p tcp -m tcp --dport 80 -j DIVERT -A DIVERT -j
>> MARK --set-xmark 0x1/0xffffffff -A DIVERT -j LOG --log-prefix "DIVERT
>> :" --log-level 7 -A DIVERT -j ACCEPT COMMIT
>>
>> ip route list table 100
>> default via 10.0.1.1 dev eth5
>> 10.0.1.0/24 dev eth5 scope link
>>
>> ip route list table main
>> defautlt via 10.1.17.254 dev eth0
>>
>>
>>
>> squid
>> eth0 10.0.1.1/24
>> gw 10.0.1.254
>>
>>
>>
>>
>>
>>
>> On 31/05/2012 16:27, Thomas York wrote:
>>> The way you have it set up (the server running Squid also being a
>>> router) works fine. I run in to problems when I try to separate them out.
>>>
>>> rp filter is disabled for all interfaces on both the proxy and the router.
>>>
>>> Here is my current network configuration for this test bed..
>>>
>>> Firewall
>>> ------------
>>> Fw (iptables router) has three network interfaces.
>>> eth0 connects to the 'internet'.
>>> eth0 has the IP of 10.1.17.158/24 with a default gateway of
>>> 10.1.17.254
>>> eth0 is being NAT'ed to allow the Windows 7 client and proxy access
>>> to the 'internet'
>>> eth1 connects to a Windows 7 client.
>>> eth1 has the IP of 10.1.1.254/24.
>>> eth5 connect to the proxy (proxy01)
>>> eth5 has the IP of 10.0.1.254/24.
>>>
>>> root_at_fw:~# ip route list table 100
>>> default via 10.0.1.1 dev eth5
>>> 10.0.1.0/24 dev eth5 scope link
>>>
>>> root_at_fw:~# iptables-save
>>> # Generated by iptables-save v1.4.13 on Thu May 31 09:23:53 2012 *raw
>>> :PREROUTING ACCEPT [1866:289737] :OUTPUT ACCEPT [788:89384] COMMIT #
>>> Completed on Thu May 31 09:23:53 2012 # Generated by iptables-save
>>> v1.4.13 on Thu May 31 09:23:53 2012 *nat :PREROUTING ACCEPT [26:3577]
>>> :INPUT ACCEPT [23:3388] :OUTPUT ACCEPT [5:352] :POSTROUTING ACCEPT
>>> [2:120] -A POSTROUTING -o eth0 -j MASQUERADE COMMIT # Completed on
>>> Thu May 31 09:23:53 2012 # Generated by iptables-save v1.4.13 on Thu
>>> May 31 09:23:53 2012 *mangle :PREROUTING ACCEPT [126:15633] :INPUT
>>> ACCEPT [126:15633] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [67:8420]
>>> :POSTROUTING ACCEPT [67:8420] :DIVERT - [0:0] -A PREROUTING -i eth2
>>> -p tcp -m tcp --dport 80 -j DIVERT -A DIVERT -j MARK --set-xmark
>>> 0x1/0xffffffff -A DIVERT -j LOG --log-prefix "DIVERT :" --log-level 7
>>> -A DIVERT -j ACCEPT COMMIT # Completed on Thu May 31 09:23:53 2012 #
>>> Generated by iptables-save v1.4.13 on Thu May 31 09:23:53 2012
>>> *filter :INPUT ACCEPT [1313:186460] :FORWARD ACCEPT [12:815] :OUTPUT
>>> ACCEPT [733:82296] COMMIT # Completed on Thu May 31 09:23:53 2012
>>>
>>> root_at_fw:~# iptables --list -t mangle
>>>
>>> Chain PREROUTING (policy ACCEPT)
>>> target prot opt source destination
>>> DIVERT tcp -- anywhere anywhere tcp dpt:http
>>>
>>> Chain INPUT (policy ACCEPT)
>>> target prot opt source destination
>>>
>>> Chain FORWARD (policy ACCEPT)
>>> target prot opt source destination
>>>
>>> Chain OUTPUT (policy ACCEPT)
>>> target prot opt source destination
>>>
>>> Chain POSTROUTING (policy ACCEPT)
>>> target prot opt source destination
>>>
>>> Chain DIVERT (1 references)
>>> target prot opt source destination
>>> MARK all -- anywhere anywhere MARK set 0x1
>>> LOG all -- anywhere anywhere LOG level
>>> debug prefix "DIVERT :"
>>> ACCEPT all -- anywhere anywhere
>>>
>>>
>>> Windows 7 client
>>> --------------
>>> The client has one network interface with the IP address of
>>> 10.1.1.253/24 and a default gateway of 10.1.1.254
>>>
>>> Proxy
>>> -------------
>>> Proxy01 (Squid proxy) has one network interface.
>>> eth0 connects to the firewall.
>>> eth0 has the IP of 10.0.1.1/24 and a default gateway of 10.0.1.254.
>>>
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Eliezer Croitoru [mailto:eliezer_at_ngtech.co.il]
>>> Sent: Wednesday, May 30, 2012 11:09 PM
>>> To: squid-users_at_squid-cache.org
>>> Subject: Re: [squid-users] Linux + TPROXY + Remote Squid
>>>
>>> i was curios about it because the last time i setup a tproxy on
>>> debian it took me couple minutes.
>>> i am using debian squeeze 6.0.5 with basic 2.6.32-5-amd64 kernel and
>>> squid
>>> 3.1.6 from debian repos and tproxy works fine for me!!
>>> debian installed with squid3 ebtables bridge-utils .
>>> (also tested with self built squid3.2.0.17 and squid3.1.19)
>>>
>>> the main thing with tproxy is to allow the VM net card promiscuous
>>> mode
>> and
>>> on the router machine disable reverse path filter using:
>>> sysctl -a |grep rp_filter
>>> should all be with the value 0
>>>
>>> i am still trying to understand what you are doing on each of the servers.
>>>
>>> what are the networks and what are the machines and what every
>>> machine
>> does?
>>>
>>> what i got until now was:
>>> W7|eth0[what ip?} -(some net)---> ethX[what ip?]--|debian_router|--[what
>>> ip?]ethX--(some net)-->ethX[what ip?]--|squid_debian|ethX[what
>>> ip?]--->{{{ internet}}}
>>>
>>> please fill my gap about ethX numbers and on any MASQUERADING that
>> happens.
>>>
>>> notice that if you are doing DNAT there is not point at all in TPROXY
>>> because the client IP was lost already.
>>>
>>> output of:
>>> iptables-save
>>> ip route list
>>> #if you are using some routing tables then also ip rotue show table
>>> table_number_or_name_here
>>>
>>>
>>>
>>>
>>> the only problem i have seen is that if i have an established session
>>> from the client and i reload the rules i get this squid error page:
>>> ##start
>>> ERROR
>>> The requested URL could not be retrieved
>>>
>>> The following error was encountered while trying to retrieve the URL:
>>> (null)://www.cnn.com/
>>>
>>> Invalid URL
>>>
>>> Some aspect of the requested URL is incorrect.
>>>
>>> Some possible problems are:
>>>
>>> Missing or incorrect access protocol (should be "http://" or
>> similar)
>>>
>>> Missing hostname
>>>
>>> Illegal double-escape in the URL-Path
>>>
>>> Illegal character in hostname; underscores are not allowed.
>>> Your cache administrator is webmaster.
>>>
>>> Generated Thu, 31 May 2012 01:17:12 GMT by localhost (squid/3.1.6)
>>> ##end
>>>
>>> i will check with the latest squid version.
>>>
>>> i am using this script to load the iptables rules:
>>>
>>> #start
>>> #!/bin/sh -x
>>> ##!/bin/sh -x
>>> #load modules requierd for the tproxy modprobe ip_tables modprobe
>>> nf_conntrack_ipv4 modprobe xt_tcpudp modprobe nf_tproxy_core modprobe
>>> xt_MARK modprobe xt_TPROXY modprobe xt_socket
>>>
>>> sysctl net.netfilter.nf_conntrack_acct sysctl
>>> net.netfilter.nf_conntrack_acct=1 ip route flush table 100 ip rule
>>> del fwmark 1 lookup 100 ip rule add fwmark 1 lookup 100 ip -f inet
>>> route
>> add
>>> local 0.0.0.0/0 dev lo table 100
>>>
>>> echo "flushing any exiting rules"
>>> iptables -t mangle -F
>>> iptables -t mangle -X DIVERT
>>>
>>> echo "creating rules"
>>> iptables -t mangle -N DIVERT
>>> iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle
>>> -A DIVERT -j ACCEPT iptables -t mangle -A PREROUTING -p tcp -m socket
>>> -j
>> DIVERT
>>> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
>>> --tproxy-mark
>>> 0x1/0x1 --on-port 3129 ##end
>>>
>>> this one for ebtables:
>>>
>>> #start
>>> #!/bin/sh -x
>>> CLIENT_IFACE="eth1"
>>> INET_IFACE="eth0"
>>>
>>> ebtables -t broute -F
>>> ebtables -t broute -A BROUTING -i $CLIENT_IFACE -p ipv4 --ip-proto
>>> tcp --ip-dport 80 -j redirect --redirect-target ACCEPT ebtables -t
>>> broute -A BROUTING -i $INET_IFACE -p ipv4 --ip-proto tcp --ip-sport
>>> 80 -j redirect --redirect-target ACCEPT
>>>
>>> ebtables -t broute -A BROUTING -i eth1 -p ipv4 --ip-proto tcp
>>> --ip-dport
>>> 80 -j redirect --redirect-target DROP ebtables -t broute -A BROUTING
>>> -i eth0 -p ipv4 --ip-proto tcp --ip-sport
>>> 80 -j redirect --redirect-target DROP
>>>
>>> cd /proc/sys/net/bridge/
>>>
>>> for i in *
>>> do
>>> echo 0> $i
>>> done
>>> unset i
>>> #end
>>>
>>> Eliezer
>>>
>>> On 25/05/2012 17:35, Thomas York wrote:
>>>> I have a lab environment set up using two Debian Wheezy servers
>>>> (Squeeze doesn't have a new enough kernel or iptables to do TPROXY
>>>> properly). One of the servers is a router and the other is a proxy
>>>> server. There are several clients connected to the router to
>>>> simulate a production routing environment. If I have both the TPROXY
>>>> redirection and Squid on the same server, Squid handles the requests
>>>> and everything works perfectly. However, this isn't how I want the
>>>> proxy to be configured in our production environment. I've changed
>>>> my iptables rules on the router to redirect all tagged 1 packets to
>>>> the proxy server. This is working perfectly fine and I can see the
>>>> data being routed to the proxy server using tcpdump on both the
>>>> router and the proxy. However, Squid on the proxy server doesn't seem to 'see'
>>>> the data being routed and doesn't do anything with it. I have
>>>> "http_port 3129 tproxy" set on the proxy server. Is there anything
>> special
>>> I need to do using iptables on the proxy server?
>>>>
>>>> Both servers are running kernel 3.2.0-2-amd64 and iptables 1.4.13
>>>> from Wheezy and the Squid being used on the proxy is 3.1.19. If any
>>>> more information is needed, please just let me know and I'd be happy
>>>> to supply it. Thanks.
>>>>
>>>> --Thomas York
>>>
>>>
>>> --
>>> Eliezer Croitoru
>>> https://www1.ngtech.co.il
>>> IT consulting for Nonprofit organizations eliezer<at> ngtech.co.il
>>
>>
>
>
> --
> Eliezer Croitoru
> https://www1.ngtech.co.il
> IT consulting for Nonprofit organizations eliezer<at> ngtech.co.il

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il