Removing multiple instances of word from dictionary files?

[jasontheblob]

i have 14 txt files which are all dictionary/wordlist files. together they are approximately 16gb, with the largest file been 2.7gb. i was wondering what ways i would be able to remove any duplicate words from the files, as i plan to merge them into one.

so basically what i am looking for it a tool/way to find multiple instances of the same word and delete it. i was thinking a tool that opens the file and moves the words into a new file, but doesnt copy across words that are already present (not sure if that is possible).

thanks alot for any help, and sorry if the above description is confusing.

[iproute]

Here is something from pure_hate http://www.question-defense.com/2010...ktrack-4-final, and there are numerous other tools capable of wordlist manipulation. John is able to manipulate wordlists, and I believe hashcat is able to also

[jasontheblob]

Here is something from pure_hate http://www.question-defense.com/2010...ktrack-4-final, and there are numerous other tools capable of wordlist manipulation. John is able to manipulate wordlists, and I believe hashcat is able to also

im not trying to manipulate the wordlist, just get rid of multiple entries of the same word. im checking out Purehates wordlist tool though.

[laptopz]

if files are sorted

uniq file > file.new

if not you`ll have to sort them first

sort file | uniq > file.new

or just

sort -u file > file.new

another way...

awk '!x[$0]++' file > file.new

[Archangel-Amael]

Topic covered. CLosed.