Results 1 to 5 of 5

Thread: Wordlist Creation

  1. #1
    Just burned his ISO
    Join Date
    Oct 2012
    Posts
    8

    Default Wordlist Creation

    I know this isn`t really a backtrack specific thread but i dont know where else to post.

    I have a 25Gb folder full of wordlists.

    I wanted to combine and clean these so i began looking into unix commands. Obviously there are the simple ones...

    Code:
    Combine:
    
    cat file1.txt file2.txt > outputfile3.txt
    ----------------------------------------------
    Sort:
    
    sort filename | uniq
    ----------------------------------------------
    Remove Duplicates:
    
    sort -u -o new_file old_file
    ----------------------------------------------
    But that still leaves alot of junk info in the final file. and takes ages to complete. It also requires me to check up on it and run the next command etc etc.

    But in my travels i found a page all about sorting and cleaning up wordlists. Removing html tags, emails etc. They gave a full run through of the commands used, but again its gonna take too much faffing about. But they did give an all-in-one set of instructions. But again, needs faffing and checking up on after each command.

    Code:
    AIO + Sort
    
        cat * > /tmp/aio-"${PWD##*/}".lst && rm * && mv /tmp/aio-"${PWD##*/}".lst ./
    
        tr '\r' '\n' < aio-"${PWD##*/}".lst > stage1-tmp && tr '\0' ' ' < stage1-tmp > stage1-tmp1 && tr -cd '\11\12\15\40-\176' < stage1-tmp1 > stage1-tmp && mv stage1-tmp stage1 && rm stage1-*
    
        htmlTags="a|b|big|blockquote|body|br|center|code|del|div|em|font|h[1-9]|head|hr|html|i|img|ins|item|li|ol|option|p|pre|s|small|span|strong|sub|sup|table|td|th|title|tr|tt|u|ul"
        cat stage1 | sed -r "s/ */ /gI;s/^[ \t]*//;s/[ \t]*$//;s/<[^>]*>//g;s/^\w.*=\"\w.*\">//;s/^($htmlTags)>//I;s/<\/*($htmlTags)$//I;s/&*/&/gI;s/"/\"/gI;s/'/'/gI;s/'/'/gI;s/</ stage2 && rm stage1
    
        sort -b -f -i -T "$(pwd)/" stage2 > stage3 && rm stage2
        grep -v " * .* " stage3 > stage3.1
        grep " * .* " stage3 > stage3.4
        rm stage3
        for fileIn in stage3.*; do
           cat "$fileIn" | uniq -c -d > stage3.0
           sort -b -f -i -T "$(pwd)/" -k1,1r -k2 stage3.0 > stage3 && rm stage3.0
           sed 's/^ *//;s/^[0-9]* //' stage3 >> "${PWD##*/}"-clean.lst && rm stage3
           cat "$fileIn" | uniq -u >> "${PWD##*/}"-clean.lst
           rm "$fileIn"
        done
        rm -f stage* #aio-"${PWD##*/}".lst
    
        wc -l "${PWD##*/}"-clean.lst
        md5sum "${PWD##*/}"-clean.lst
    What i want to do is turn this into a full script i can just run and have it do all the commands one after another and give me a final result. Prefferably a script that i can just point to the folder and run. But i have no idea about scripting and wondered if there is anyone out there that could help me???

    The source for this is here

  2. #2
    Just burned their ISO
    Join Date
    Dec 2012
    Posts
    2

    Default Re: Wordlist Creation

    Hey there I am glad I came across your post I am currently working on a script to do just that. I wasn't planning on having it perform all of these actions but think the source you linked to will allow me to expand on my original idea. Now I too am new to scripting and am just now learning the language but this is the project I have been working on as I learn. Though I am not sure when I will be done I will certainly post for everyone to use if necessary. Meanwhile if you are interested in learning to write your own script there are a ton of tutorials and references online or by book. Thank you for posting the link this will certainly help me layout the format I was looking for!

  3. #3
    My life is this forum thorin's Avatar
    Join Date
    Jan 2010
    Posts
    2,629

    Default Re: Wordlist Creation

    You should checkout TAPE's blog he has tons of posts on wordlist creation and manipulation:

    http://adaywithtape.blogspot.ca/

    There's usually only one post showing on the main page so be sure to check the historic or archive bits in the right-nav.
    I'm a compulsive post editor, you might wanna wait until my post has been online for 5-10 mins before quoting it as it will likely change.

    I know I seem harsh in some of my replies. SORRY! But if you're doing something illegal or posting something that seems to be obvious BS I'm going to call you on it.

  4. #4
    Senior Member
    Join Date
    Jul 2010
    Location
    UK
    Posts
    136

    Default Re: Wordlist Creation

    There was a script by purehate for wordlist manipulation, I think it was more based on file changes rather than creating wordlists etc.

    I used to use it and was very helpful;
    http://www.backtrack-linux.org/forum...read.php?t=689
    Last edited by Jimmy87; 01-15-2013 at 10:15 AM. Reason: added link

  5. #5
    Very good friend of the forum TAPE's Avatar
    Join Date
    Jan 2010
    Location
    Europe
    Posts
    599

    Default Re: Wordlist Creation

    Have a look through WLM script and see if anything there can help you out ;

    wlm_v0-7.jpg

    http://code.google.com/p/wordlist-ma...wiki/WLM_USAGE

    http://adaywithtape.blogspot.nl/2012...-with-wlm.html
    Last edited by TAPE; 01-17-2013 at 01:24 AM.

Similar Threads

  1. Wordlist Generator Script - Revamping Original Wordlist
    By mcurran in forum OLD Programming
    Replies: 5
    Last Post: 01-22-2010, 06:13 AM
  2. [Moved] Wordlist creation question
    By im-a-skier in forum OLD Newbie Area
    Replies: 8
    Last Post: 11-17-2009, 09:32 PM
  3. Replies: 2
    Last Post: 11-25-2008, 11:42 AM
  4. Replies: 28
    Last Post: 10-23-2008, 10:28 AM
  5. iso creation
    By xvbanevx in forum OLD Newbie Area
    Replies: 3
    Last Post: 05-01-2007, 06:11 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •