Re: [squid-users] Squid on DualxQuad Core 8GB Rams - Optimization - Performance - Large Scale - IP Spoofing

Haytham KHOUJA (devnull) wrote:
> Hello,
> The purpose of this thread is to join forces to have the best Squid
> configuration for generic affordable Intel machines available by major
> vendors (Dell/HP...) specifically for ISPs and corporations that want a
> basic setup but with optimal response and throughput and maximizing
> bandwidth savings.
> I work for an important ISP and I currently replaced 2 NetApp NetCache
> with 3 Dell 2950 hooked up on a Foundry Switch for Load Balancing.
> I used tproxy to enable IP Spoofing to IP spoofing the outgoing address
> with some configurations on the Cisco core router, I had to compile
> iptables and tproxy on a Debian kernel source (2.6.18)
>
> I've read almost every single thread on Optimizing Squid and Linux and
> want to share my setup with you.
> I do have some questions, clarifications and bugs but overall the
> performance is pretty impressive. (Yes, much better than the NetApps)
>
> What i want to do is since i have 8 GB of RAMs, i want to store more hot
> objects in the RAMs to maximize Memory hit ratio, but with my setup,
> Squid doesn't
> go above 2GB~3GB of usage. (Remember, that there are no other heavy
> processes on the machine).

You will need a 64-bit enabled squid to go higher than 2GB.

>
> If i knew beforehand that Squid doesn't make use of SMP, i wouldn't have
> bought Dual Quad Core and would have invested in Intel CPUs with 8mb of
> Cache, but what's done is done :)
>
> Before i had Squid go down because of File Delimiters and maximum open
> files and ip_conntrac fill up, i fixed both with some iptables and
> sysctl configuration.
> Now i'm hitting a "Oct 14 01:17:06 proxy4 squid[8883]: assertion failed:
> diskd/store_io_diskd.c:384: "!diskdstate->flags.close_request" Error, so
> Squid kills and restarts (which flushes the Memory cache).
>
> I'm looking forward for some contributions, idea sharing, knowledge
> correcting to make this setup a standard setup for large scale, well
> optimized and high performant Squid for future tweakings. I hope this
> configuration would be then uploaded to the Squid wiki.

Post your squid.conf to
   http://squid.treenet.co.nz/cf.check/
and review the results. I've pointed out the biggest worries below.

> Here's my setup:
> Dell 2950
> Dual Quad Core 2.4Ghz / 8 GB Rams / 4x 136 GB 15000 RPM drives
>
> I have 3 cache_dir on separate drives and I formated the 3 disks with
> ReiserFS:
> /dev/sdb1 /CACHE1 reiserfs notail,noatime 0 0
> /dev/sdc1 /CACHE2 reiserfs notail,noatime 0 0
> /dev/sdd1 /CACHE3 reiserfs notail,noatime 0 0
>
> I run Debian GNU/Linux Etch and compiled Squid with the following:
> Squid Cache: Version 2.6.STABLE16
> configure options: '--bindir=/usr/bin' '--sbindir=/usr/sbin/'
> '--sysconfdir=/etc' '--enable-icmp' '--enable-snmp' '--enable-async-io'
> '--enable-linux-netfilter' '--enable-linux-tproxy' '--with-dl'
> '--with-large-files' '--enable-large-cache-files' '--with-maxfd=1000000'
> '--enable-storeio=diskd,ufs' '--with-aio' '--enable-epoll'
> '--disable-ident-lookups' '--enable-removal-policies=heap'
> 'CFLAGS=-DNUMTHREADS=120'
>
> As you can see i have the following modules enabled: linux-tproxy,
> diskd, epoll, and removal policies.
> /dev/epoll improves network I/O performance, Diskd separates disk I/O to
> separate processes (which reduces process locking from Squid to write on
> disks), and read benchmarks for memory and disk removal policies.

aufs does a better job, particularly where threads are available and is
not quite so broken as diskd.

> My /etc/squid.conf is composed of the following:
>
> http_port 80 transparent tproxy
> tcp_outgoing_address IP of the Machine
> :: Those are for IP Spooding and Transparency
>
> via off
> forwarded_for off
> :: Those are for total transparency, remote hosts will never guess that
> the request came from a proxy

IIRC, theres more than this needed for complete silence. They just
replace the Via and Forwarded-For with text 'unknown'. still leaving the
headers in place for anon-proxy identification.

>
> cache_mem 600 MB
> :: A bit confused about this, When i go higher than 2GB, Squid kills
> with a "out of memory" error. I have 8GB and want to maximize the use of
> it.
>
> cache_effective_user nobody
> cache_effective_group nogroup
> :: Security and bla bla

This is the default UID. If this is going to be a standard config these
MUST not be explicitly set.
Also when GID is configured as above, will in fact cause a
squid-specific deviation from the configured OS-level security policy.

They are no longer to be used, unless the machine-specific setup
requires it AND the admin knows how to setup for them properly.

> cache_replacement_policy heap LFUDA
> memory_replacement_policy heap GDSF
> :: Very objective, you can google about them
>
> cache_dir diskd /CACHE1 61440 16 256 Q1=144 Q2=128
> cache_dir diskd /CACHE2 61440 16 256 Q1=144 Q2=128
> cache_dir diskd /CACHE3 61440 16 256 Q1=144 Q2=128
> :: DISKD configuration, i'm only using 60GB of each disk
>
> cache_access_log /var/log/squid/access.log

Obsolete option. Use access_log with same parameters instead.

> cache_log /var/log/squid/cache.log
> cache_store_log none
> :: No need to log cache_store, so minimizing the Disk I/O
>
> fqdncache_size 51200
> ipcache_size 51200
> :: Caching IPs/Domain Name and whatnot
>
> pipeline_prefetch on
> :: Performance enhancement
>
> shutdown_lifetime 1 second
> :: Tired to wait whenever i restart my Squids (Only on testing)
>
> read_ahead_gap 60 KB
> maximum_object_size 2 GB
> minimum_object_size 0 KB
> maximum_object_size_in_memory 128 KB
> cache_swap_high 80%
> cache_swap_low 70%
> half_closed_clients off
> memory_pools on
> positive_dns_ttl 24 hours
> negative_dns_ttl 30 seconds
> request_timeout 60 seconds
> connect_timeout 30 seconds
> pconn_timeout 30 seconds
> ie_refresh on
> dns_nameservers DNS1 DNS2
> emulate_httpd_log off
> log_ip_on_direct on
> debug_options ALL, 9

performance enhancements above to minimize disk IO yet you log
everything at full-debug? this *,9 could cause extremely high disk usage
under load. Try *,1 (minimal) or *,5 (detailed overview) instead.

> pid_filename /var/run/squid.pid
>
> My IPtables/sysctl and startup file:
> #!/bin/sh
> iptables -t tproxy -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j
> TPROXY --on-port 80
> :: I run Squids on port 80 so that i can forward all incoming requests
> on port 80 to the Squids on the Cisco router level
>
> echo 1 > /proc/sys/net/ipv4/ip_forward
> echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
> echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
> echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range
> echo 102400 > /proc/sys/net/ipv4/tcp_max_syn_backlog
> echo 1000000 > /proc/sys/net/ipv4/ip_conntrack_max
> echo 1000000 > /proc/sys/fs/file-max
> echo 60 > /proc/sys/kernel/msgmni
> echo 32768 > /proc/sys/kernel/msgmax
> echo 65536 > /proc/sys/kernel/msgmnb
> :: Maximizing Kernel configuration
>
> ulimit -HSn 1000000
> /etc/init.d/squid stop
> /etc/init.d/squid start
> :: Re-enforcing ulimit parameters for the Squid process.
>
> Thank you

No, thank you.

Amos
Received on Sun Oct 14 2007 - 07:14:03 MDT