Logging http traffic

[Phoneywar]

Hi all,

Yet another Lurker take the plunge, (hmm... must remember to return that )

Seriously, I've been lurking here for several months on and off. It seems a fairly friendly place, except for the Idiots Corner of course. I do enjoy reading some of the nonsense which goes on in there

I'm not sure that this is the right place but as security professionals it is in your field of expertise so I'm hoping you may be able to suggest something even if it is don't waste anymore time on this.

I've been looking a tool which will allow me to securely capture http traffic. The basic criteria are as follows:

* Secure logging of traffic.
* Restricted access.
* Automatic purging of expired data without human intervention.
* The ability to examine and/or extract specific data without seeing unrelated data.

For example, if xyz complains someone from here did whatever on or to their website. I could log into the 'system' and run a search for xyz.com at the specified date and time. This would give me a list of requests along with date, time and local ip. From there I could examine the request, the http headers and the server response etc. to see what actually went on.

I suspect that a database management system is the most suitable option for the 'backend' but actually capturing the data has me stumped. Apart from spam-vertised crap the only thing I've been able to find which performs a similar function is a http capture proxy called Paros which is a diagnostic tool and simply not suitable.

Before anyone starts screaming about unlawful interception please try actually reading the RIP act. No, on second thoughts don't do that. I don't wish to be responsible for anyone suffering brain damage as a result of my advice

Seriously, Please accept my assurance that it is not unlawful as I am the sole and undiluted owner of the hardware in question and pay for the internet access. I am aware of the privacy issues which is why the thing needs to be secure and have the ability to automatically purge expired entries without anyone ever having seen them.


I was also intending to contribute to cormega's thread on his honeynet project as well but that will have to wait I guess.

[purehate]

well there are actually several options:

What I do is have a firewall box which is dedicated and has 2 nic cards so all traffic passes through the box. so its set up like this

Web > modem > firewall > router > lan

Then the 2 nics are bridged so that there is only one as far as the computer knows. Then I can use the box to run tcpdump or whatever logging program I want on that interface. You can then write a simple program and database to log only http traffic and write it to a file or database and have it clean itself out every week or month or whatever. Most firewall distros also have graphical tools in them to monitor traffic and such.

The other way is to have a IDS box and use a tap which is basically 4 rj45 connector set up in such a way that the line is "taped". I don't have one of these but several people on the forum use them.

A third option would be to use a third party firmware like ddwrt or openwrt on a linksys router and use that as your gateway if were talking about a small network. You can run the same tcpdump on one of those and write the file to a desktop computer or some thing.

Hope that helps.

[streaker69]

You can also use Squid as a transparent proxy and setup SARG as a reporting agent that reads Squid log files. That's about the easiest way to do it.

[Phoneywar]

A dedicated box sitting between the modem and the switch is precisely what I was planning.

I did consider tcpdump but it's not really suitable. A 'transparent' proxy is more along the lines I was thinking of, particularly after testing Paros with firefox.

I'll have another look at squid, along with SARG this time, but I suspect I'm going to have to adapt an existing proxy to do the job. That may take a while...

Thanks for your help.