Results 1 to 7 of 7

Thread: Best tool for capturing http requests en masse

  1. #1
    Just burned their ISO
    Join Date
    Apr 2010
    Posts
    16

    Default Best tool for capturing http requests en masse

    I am working on a project where the client needs to capture all http requests (top level domain) of all sites surfed to by their employees. Typical log transactions include not only the top level domain, but subsequent calls to links within that domain, of which the client is NOT interested in capturing. The capturing will be done on a very active lan so I think ettercap may not be able to handle the quantity of requests, and arp poisoning is not an option.

    I'm looking for a tool that will be able to capture the http request and then filter out and save ONLY the domain names. For example:

    hxxp://www.ntimes.com ----> would like to capture this, but NOT by the same user:

    hxxp://www.nytimes.com/graphics/stories/9789.jpg


    Does anyone one of a tool like this? Would it be included in backtrack?
    Last edited by bulgin; 08-06-2010 at 08:11 PM. Reason: Removed url link

  2. #2
    Good friend of the forums gunrunr's Avatar
    Join Date
    Jan 2010
    Location
    shining my spoon
    Posts
    265

    Default Re: Best tool for capturing http requests en masse

    how about the gateway router that your clients network is using, most of them can do that for you, they will keep records of that info ant the internal ip addresses which are sending those get rquests
    Wielder of the spoon of doom
    Summercon, Toorcon, Defcon, Bsides, Derbycon, Shmoocon oh my
    Come hang out with hackers on twitter @gunrunr556

  3. #3
    Just burned their ISO
    Join Date
    Apr 2010
    Posts
    16

    Default Re: Best tool for capturing http requests en masse

    That is a possibility. Problem is the get requests oftentimes contain multiple get requests within the same domain for a single request, and I have to come up with a way to filter out all the associated get requests for the requesting IP. The example:

    hxxp://www.ntimes.com ----> would like to capture this, but NOT by the same user:

    hxxp://www.nytimes.com/graphics/stories/9789.jpg

    explains the problem. I was hoping there is a program within backtrack or a program that members of this forum may know, that easily can strip out all requests other than the primary, first get request to that page.

    I may have to make a mash up of something for that I fear....

  4. #4
    Senior Member
    Join Date
    Jul 2009
    Posts
    135

    Default Re: Best tool for capturing http requests en masse

    a tcpdump filter that captures only DNS requests/data from the client (no data on server end will be captured). Here i took the liberty of doing this for you.

    # tcpdump -lni eth0 'dst port 53'|cut -d " " -f 8

    The above will give you the requested URLs.

    # tcpdump -lni eth0 'dst port 53'|cut -d " " -f 3

    The above will give you the requesting host's IP address.
    Last edited by aerokid240; 08-31-2010 at 01:40 PM.

  5. #5
    My life is this forum thorin's Avatar
    Join Date
    Jan 2010
    Posts
    2,629

    Default Re: Best tool for capturing http requests en masse

    I haven't played with it yet but the tool discussed here might be able to help you:
    Web Traffic Analysis with httpry

    aerokid240 has a good idea as well. Chances are high that very little of the DNS traffic from the LAN is not HTTP/HTTPS related, you could profile it pretty quickly in a few days and find out the percentages. Simply logging all the DNS look-ups would give the details you're after. There might be a bit that is IM related, email related, FTP related, but the I'd be surprised if the overwhelming majority isn't HTTP/HTTPS related.

    Also if you're finding wireshark/ettercap/tcpdump too slow you could check gulp:
    http://staff.washington.edu/corey/gulp/gulpman.html

    Lastly you'd have to do some research but you could probably also drop a SNORT box or something similar in-line and use it for logging what you're looking for.
    Last edited by thorin; 08-27-2010 at 07:22 PM.
    I'm a compulsive post editor, you might wanna wait until my post has been online for 5-10 mins before quoting it as it will likely change.

    I know I seem harsh in some of my replies. SORRY! But if you're doing something illegal or posting something that seems to be obvious BS I'm going to call you on it.

  6. #6
    My life is this forum thorin's Avatar
    Join Date
    Jan 2010
    Posts
    2,629

    Default Re: Best tool for capturing http requests en masse

    URLSnarf can probably help you as well, it's part of the dsniff collection.
    Last edited by thorin; 08-31-2010 at 12:38 PM.
    I'm a compulsive post editor, you might wanna wait until my post has been online for 5-10 mins before quoting it as it will likely change.

    I know I seem harsh in some of my replies. SORRY! But if you're doing something illegal or posting something that seems to be obvious BS I'm going to call you on it.

  7. #7
    Just burned his ISO DrWho's Avatar
    Join Date
    Aug 2010
    Location
    Left Coast, USA
    Posts
    3

    Default Re: Best tool for capturing http requests en masse

    Seems like EtherPeek (and AiroPeek for WiFi) or WireShark would fill the bill. With EtherPeek in particular, you can set up your filters to grab exactly what you're trying to do. I've used it to do similar tasks at my old job. You can set up the filters any way you want like, showing all the requesting IPs only once and every destination IP they're requesting but not the gets and acks. Or Vise-Versa. Showing every destination IP and who requested it, the connection duration, etc. Then you can export it to several formats.
    You can save the entire capture and generate different reports based on the information you want. The capture is everything it was listening to. The filters are applied AFTER the capture, not during as many sniffers do. I like to use it for capturing passwords used by automated processes our software creators forget to document and can't remember.
    We had about 1500 users on site (26000+ across the nation) and EtherPeek never gave me a problem with the throughput (an OC-9, OC-12 & OC-192 all coming in at the same MPOE! We had more bandwidth than the local PacBell CO!) and never missed a packet. It's very useful. WireShark can do roughly the same, but I don't use it often enough to say exactly what it can and can't do.
    Hope this helps.

Similar Threads

  1. Ettercap not capturing HTTP or HTTPS
    By falseteeth in forum Beginners Forum
    Replies: 1
    Last Post: 08-04-2010, 03:45 AM
  2. Replies: 0
    Last Post: 01-11-2010, 08:09 AM
  3. Tool requests - NSAT, finger, NFS, xwatchwin, winlockpwn
    By williamc in forum OLD BT4 Feature Requests
    Replies: 1
    Last Post: 06-02-2009, 03:08 AM
  4. Tool requests
    By williamc in forum OLD BT3beta General
    Replies: 2
    Last Post: 06-19-2008, 08:39 AM
  5. HTTP Session ID brute forcing tool?
    By imported_Tr8shCan in forum OLD Specialist Topics
    Replies: 0
    Last Post: 04-10-2008, 02:06 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •