Comparing Wordlists and REMOVING Words Without Combining Them

[Sulegna]

Hey Folks,

I've tried searching for this, but not sure what kind of keywords I would type to find such a thing.

Is there a script, or even simpler a command line, that allows you to compare two word lists and REMOVE words from one of them?

For example, compare WorldList_01 to WordList_02 and remove from WordList_02 words that are in WordList 01, leaving you with two wordlists that are unique to each other?

My problem is that I have two wordlists that I have created.
One is a list of dates starting from 01 01 1900 (with no spaces of course) and going all the way up to 01 01 2013. I have them in 6 variations:
mmddyyyy
mmyyyydd
ddyyyymm
ddmmyyyy
yyyymmdd
yyyyddmm

I also have it "uniqued" since 01011900 can be both mmddyyyy and ddmmyyyy. (If anyone wants this date list, let me know.)

The other is a list of 8-character permutated numbers using crunch from 00000000 to 99999999. This file is about 900 megabytes big.

Now, of course, the crunch-generated list will have every single date in my date list. I would like to remove all of those dates from my date list from my crunch-generated list so that the two lists are unique compared to each other. Is there a way to do this?

Many thanks in advance!

- Sulegna -

[Archangel-Amael]

compare two word lists in google
How do I Compare two files under Linux or UNIX?

cat wordlist.lst | sort | uniq | merge > newwordlist.lst
See also http://www.backtrack-linux.org/forum...eneration.html
This has pretty much everything you need to know.

http://www.backtrack-linux.org/forum...tionaries.html
http://www.backtrack-linux.org/forum...ords-john.html



Again Topic is covered.

[Sulegna]

Hey Folks,

Okay, to AVOID CONFUSION, I am going to attempt to reword this post so that it is clear as to what I am searching for.

I have HONESTLY tried searching for this, but I cannot find a simple way to do this.

I would like to compare one wordlist to another, and remove words from the second wordlist WITHOUT having to combine the two lists together and using sort & uniq to remove the duplicates. I already know how to combine multiple wordlists and use sort and uniq to remove all the duplicates. This is NOT what I am trying to do. I want to keep the two files SEPARATE.

For example, compare WorldList_01 to WordList_02 and remove from WordList_02 words that are in WordList_01, leaving you with TWO unique wordlists.

My problem is that I have two wordlists that I have created.
One is a list of dates starting from 01 01 1900 (with no spaces of course) and going all the way up to 01 01 2013. I have them in 6 variations:
mmddyyyy
mmyyyydd
ddyyyymm
ddmmyyyy
yyyymmdd
yyyyddmm

I also have it "uniqued" since 01011900 can be both mmddyyyy and ddmmyyyy. (If anyone wants this date list, let me know.)

The other is a list of 8-character permutated numbers using crunch from 00000000 to 99999999. This file is about 900 megabytes big.

Now, of course, the crunch-generated list will have every single sequence of numbers in my date list. What I would like to do is remove everything listed in my date list from my crunch-generated list, WITHOUT combining the two of them, so that the two separate lists are unique compared to each other.

Is there a way to do this?

Many thanks in advance!

- Sulegna -

[Archangel-Amael]

Linux / UNIX Display Lines Common in Two Files
Comparing 2 files and return the unique lines in first file - The UNIX and Linux Forums
awk updating one file with another, comparing, updating - The UNIX and Linux Forums
Compare lists of files - The UNIX and Linux Forums
Compare two txt files line by line | Planet Admon

Google

Now, again this is the last time I will tell you to search before making a post here. I have tried to be nice, and honestly I have put up with this longer than I should. The links above will either cover or put you in close proximity to your goal.
This topic is covered and as such closed.