Results 1 to 5 of 5

Thread: Removing multiple instances of word from dictionary files?

  1. #1
    Just burned his ISO
    Join Date
    Jan 2011
    Posts
    2

    Default Removing multiple instances of word from dictionary files?

    i have 14 txt files which are all dictionary/wordlist files. together they are approximately 16gb, with the largest file been 2.7gb. i was wondering what ways i would be able to remove any duplicate words from the files, as i plan to merge them into one.

    so basically what i am looking for it a tool/way to find multiple instances of the same word and delete it. i was thinking a tool that opens the file and moves the words into a new file, but doesnt copy across words that are already present (not sure if that is possible).

    thanks alot for any help, and sorry if the above description is confusing.

  2. #2
    Senior Member iproute's Avatar
    Join Date
    Jan 2010
    Location
    Midwest, USA
    Posts
    192

    Default Re: Removing multiple instances of word from dictionary files?

    Here is something from pure_hate http://www.question-defense.com/2010...ktrack-4-final, and there are numerous other tools capable of wordlist manipulation. John is able to manipulate wordlists, and I believe hashcat is able to also
    Last edited by iproute; 01-17-2011 at 08:09 PM.

  3. #3
    Just burned his ISO
    Join Date
    Jan 2011
    Posts
    2

    Default Re: Removing multiple instances of word from dictionary files?

    Quote Originally Posted by iproute View Post
    Here is something from pure_hate http://www.question-defense.com/2010...ktrack-4-final, and there are numerous other tools capable of wordlist manipulation. John is able to manipulate wordlists, and I believe hashcat is able to also
    im not trying to manipulate the wordlist, just get rid of multiple entries of the same word. im checking out Purehates wordlist tool though.

  4. #4
    Junior Member laptopz's Avatar
    Join Date
    Dec 2010
    Posts
    55

    Default Re: Removing multiple instances of word from dictionary files?

    if files are sorted
    Code:
    uniq file > file.new
    if not you`ll have to sort them first
    Code:
    sort file | uniq > file.new
    or just
    Code:
    sort -u file > file.new
    another way...
    Code:
    awk '!x[$0]++' file > file.new
    If anything can go wrong, it will....

  5. #5
    Super Moderator Archangel-Amael's Avatar
    Join Date
    Jan 2010
    Location
    Somewhere
    Posts
    8,012

    Default Re: Removing multiple instances of word from dictionary files?

    Topic covered. CLosed.
    To be successful here you should read all of the following.
    ForumRules
    ForumFAQ
    If you are new to Back|Track
    Back|Track Wiki
    Failure to do so will probably get your threads deleted or worse.

Similar Threads

  1. Some noob questions about dictionary files
    By StoneFox in forum Beginners Forum
    Replies: 9
    Last Post: 07-12-2010, 02:59 PM
  2. Converting Dictionary files?
    By imcookie in forum OLD Newbie Area
    Replies: 0
    Last Post: 03-29-2010, 10:48 PM
  3. how to remove junk chracters from word list files
    By benjsh in forum OLD Newbie Area
    Replies: 5
    Last Post: 08-11-2009, 03:06 PM
  4. Dictionary Files length
    By Pirates in forum OLD Newbie Area
    Replies: 6
    Last Post: 12-05-2007, 01:38 PM
  5. Word Lists - Dictionary lists??
    By wylde342 in forum OLD Newbie Area
    Replies: 6
    Last Post: 07-06-2007, 05:45 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •