Results 1 to 4 of 4

Thread: Comparing Wordlists and REMOVING Words Without Combining Them

  1. #1
    Just burned his ISO
    Join Date
    Sep 2010
    Posts
    12

    Question Comparing Wordlists and REMOVING Words Without Combining Them

    Hey Folks,

    I've tried searching for this, but not sure what kind of keywords I would type to find such a thing.

    Is there a script, or even simpler a command line, that allows you to compare two word lists and REMOVE words from one of them?

    For example, compare WorldList_01 to WordList_02 and remove from WordList_02 words that are in WordList 01, leaving you with two wordlists that are unique to each other?

    My problem is that I have two wordlists that I have created.
    One is a list of dates starting from 01 01 1900 (with no spaces of course) and going all the way up to 01 01 2013. I have them in 6 variations:
    mmddyyyy
    mmyyyydd
    ddyyyymm
    ddmmyyyy
    yyyymmdd
    yyyyddmm

    I also have it "uniqued" since 01011900 can be both mmddyyyy and ddmmyyyy. (If anyone wants this date list, let me know.)

    The other is a list of 8-character permutated numbers using crunch from 00000000 to 99999999. This file is about 900 megabytes big.

    Now, of course, the crunch-generated list will have every single date in my date list. I would like to remove all of those dates from my date list from my crunch-generated list so that the two lists are unique compared to each other. Is there a way to do this?

    Many thanks in advance!

    - Sulegna -

  2. #2
    Super Moderator Archangel-Amael's Avatar
    Join Date
    Jan 2010
    Location
    Somewhere
    Posts
    8,012

    Default Re: Comparing Wordlists and REMOVING Words Without Combining Them

    compare two word lists in google
    How do I Compare two files under Linux or UNIX?

    cat wordlist.lst | sort | uniq | merge > newwordlist.lst
    See also http://www.backtrack-linux.org/forum...eneration.html
    This has pretty much everything you need to know.

    http://www.backtrack-linux.org/forum...tionaries.html
    http://www.backtrack-linux.org/forum...ords-john.html



    Again Topic is covered.
    Last edited by Archangel-Amael; 09-04-2010 at 03:12 PM.
    To be successful here you should read all of the following.
    ForumRules
    ForumFAQ
    If you are new to Back|Track
    Back|Track Wiki
    Failure to do so will probably get your threads deleted or worse.

  3. #3
    Just burned his ISO
    Join Date
    Sep 2010
    Posts
    12

    Default Comparing 2 Wordlists and Removing Identical Words WITHOUT Combining The Lists

    Hey Folks,

    Okay, to AVOID CONFUSION, I am going to attempt to reword this post so that it is clear as to what I am searching for.

    I have HONESTLY tried searching for this, but I cannot find a simple way to do this.

    I would like to compare one wordlist to another, and remove words from the second wordlist WITHOUT having to combine the two lists together and using sort & uniq to remove the duplicates. I already know how to combine multiple wordlists and use sort and uniq to remove all the duplicates. This is NOT what I am trying to do. I want to keep the two files SEPARATE.

    For example, compare WorldList_01 to WordList_02 and remove from WordList_02 words that are in WordList_01, leaving you with TWO unique wordlists.

    My problem is that I have two wordlists that I have created.
    One is a list of dates starting from 01 01 1900 (with no spaces of course) and going all the way up to 01 01 2013. I have them in 6 variations:
    mmddyyyy
    mmyyyydd
    ddyyyymm
    ddmmyyyy
    yyyymmdd
    yyyyddmm

    I also have it "uniqued" since 01011900 can be both mmddyyyy and ddmmyyyy. (If anyone wants this date list, let me know.)

    The other is a list of 8-character permutated numbers using crunch from 00000000 to 99999999. This file is about 900 megabytes big.

    Now, of course, the crunch-generated list will have every single sequence of numbers in my date list. What I would like to do is remove everything listed in my date list from my crunch-generated list, WITHOUT combining the two of them, so that the two separate lists are unique compared to each other.

    Is there a way to do this?

    Many thanks in advance!

    - Sulegna -

  4. #4
    Super Moderator Archangel-Amael's Avatar
    Join Date
    Jan 2010
    Location
    Somewhere
    Posts
    8,012

    Default Re: Comparing 2 Wordlists and Removing Identical Words WITHOUT Combining The Lists

    Linux / UNIX Display Lines Common in Two Files
    Comparing 2 files and return the unique lines in first file - The UNIX and Linux Forums
    awk updating one file with another, comparing, updating - The UNIX and Linux Forums
    Compare lists of files - The UNIX and Linux Forums
    Compare two txt files line by line | Planet Admon

    Google

    Now, again this is the last time I will tell you to search before making a post here. I have tried to be nice, and honestly I have put up with this longer than I should. The links above will either cover or put you in close proximity to your goal.
    This topic is covered and as such closed.
    To be successful here you should read all of the following.
    ForumRules
    ForumFAQ
    If you are new to Back|Track
    Back|Track Wiki
    Failure to do so will probably get your threads deleted or worse.

Similar Threads

  1. Replies: 0
    Last Post: 09-04-2010, 07:58 AM
  2. Comparing Two Wireless Cards
    By Mortifix in forum OLD Newbie Area
    Replies: 1
    Last Post: 01-19-2010, 05:56 PM
  3. combining words in dictionaries
    By the fool in forum OLD Newbie Area
    Replies: 2
    Last Post: 01-17-2010, 06:00 PM
  4. Combining words in John
    By Sub101 in forum OLD Wireless
    Replies: 4
    Last Post: 09-01-2008, 12:41 PM
  5. Combining Airodumps
    By Spyder_Snyper in forum OLD Newbie Area
    Replies: 2
    Last Post: 05-14-2007, 08:20 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •