Re: [squid-users] Looking for web usage reporting solution

On Fri, Nov 13, 2009 at 11:31 AM, Aaron Spurlock
<aarons_at_technovationdesign.com> wrote:
> I am looking for a web usage reporting solution that can run via sniffing or from a mirror port on a switch. I envision this solution would simply log each URL request it sees and allow reports to be generated on web sites that internal users have gone to.  I've searched high and low, but cannot find a "ready-made" solution, so I'm looking to put it together myself.
>
> Most people/posts suggest using squid/squidgard/dan's guardian, but it appears to me that is only an inline solution, and I would prefer a sniffing solution for safety (if machine crashes, it doesn't take down Internet). In that sense, it would work a lot like websense, but without the blocking, only reporting.
>
> From a high-level pseudo-code standpoint, it would simply sniff all traffic, and when it sees a packet requesting a webpage, it parses it and dumps these results into a database:
>
> -Date
> -Time
> -Source IP
> -Dest IP
> -URL requested
> -FQDN portion of web request - IE: if request was for
>  http://www.microsoft.com/windows/server/2003, it records only
>  www.microsoft.com here
> -domain portion of web request - only microsoft.com in above example
>
> Using this data, I can then produce reports for the client on who went where when.... Personally, I thought this would be a great program for open source, but I can't find anything like this already out there!!! It seems like kind of a mix between Squid, NTOP and Snort...
>
> Thanks for any thoughts on this project!
>

Sounds like you've got it pretty much worked out...not sure what the
question is? Sniffing for HTTP traffic should work fine, then you just
need to parse the traffic. I would probably just use iptables to sniff
for traffic on port 80 and 8080. Of course, you can't sniff SSL
communications, but there's really no solution to that unless you can
break the encryption (which you can't). I guess this is technically
still an inline solution, but short of actually buying a hardware
packet sniffer, I'm not sure that there is any "out of line" solution.

-Brian

-- 
Feel free to contact me using PGP Encryption:
Key Id: 0x3AA70848
Available from: http://keys.gnupg.net