# Removing patterns from Numeric wordlist

Show 40 post(s) from this thread on one page
Page 2 of 2 First 12
• 05-04-2012, 10:53 PM
Re: Removing patterns from Numeric wordlist
very good point gitsnik, my order was a bit out of whack and there really isn't the need to pipe it so many times. I'm interested to see how large the final product would be and how much all these computations would slow down your crunching.

I think I figured out what I'm doing this afternoon ;)
• 05-05-2012, 06:03 AM
Gitsnik
Re: Removing patterns from Numeric wordlist
Haha no probs, hit TAPE up if you run in to any issues, he can contact me if he needs the extra and looks like you two could come up with the answers properly anyway. Look forward to seeing it.

• 05-05-2012, 09:04 PM
Re: Removing patterns from Numeric wordlist
haha, whoops I just added more crap too. wow I suck
• 05-07-2012, 08:12 AM
thorin
Re: Removing patterns from Numeric wordlist
Quote:

Originally Posted by SomeNewbThatFeelsEntitled
So please, if you are one of the few who have already posted here, kindly stay away from my thread.

It belongs to an open community of volunteers that owe you absolutely nothing. If you wish this thread to be yours please feel free to spin up your own domain and server, start your own forums with no users, and post your question.

Quote:

Numeric, and otherwise, wordlists have patterns that are unlikely to be utilized by software which creates the key based on 'random' digits. Repetition is an obvious one, but all patterns must go.

0000000001
0101010101
89abcdef
febf2cff (utilizing the 'f' 4/8 makes this key highly improbable to be generated randomly)
You seem to be operating under a false premiss. Repetition does not denote any failure of randomness. In fact removing seeming patterns and repetition simply removes valid outcomes. If a value is truly random then it's just as likely to contain repetition (00000) vs all unique values (12345). The fact that your 4th example contains 4fs out of 8 does not make it any more or less likely to be generated at random.

The same applies for sequences and other patterns. If you randomly generate values you'll eventually see words (in any and all languages even) that doesn't render the value non-random. It simply means that someone interprets them as having meaning or a pattern.

Consider the case for dual factor authentication and use of key fobs (i.e.: RSA tokens) if the developer(s) used some kind of pattern recognition on the generated values and dropped all duplicates or sequential values they'd simply be reducing the number of values an attacker has to attempt in order to hit a correct "randomly generated" one-time-password.
• 05-15-2012, 03:11 AM
Re: Removing patterns from Numeric wordlist
I just finished making a customizable script that accomplishes all this and allows you to customize:

mid and max string length
output filename / directory
the max number of consecutive characters allowed
the max number of one character allowed
the charset that will be crunched

I'll try to post it here tomorrow or tonight at the earliest. I was planning on adding it to my word list toolkit but I don't think i'm quite ready to release v1.1 just yet.
• 05-15-2012, 06:59 PM
Re: Removing patterns from Numeric wordlist
On the risk of adding more crap to this thread, I'd just like to say that while I look forward to adding thad0ctor's tool to my arsenal, what is this adding that a well-written JtR rule would not have? The semblance of taking away multiple chars? ...or what?
• 05-16-2012, 03:52 AM
Re: Removing patterns from Numeric wordlist
I'm pretty sure you could do the same with JTR rules but I'm just not familiar with JTR at all. Instead my script uses most bash and crunch to get the job done. In addition, you can customize everything from:

the min and max password length
the charset
the maximum number of sequenctial characters
and the max number of one character per line created

everything works great but it really slows down crunch. granted I'm using it on a 2 core laptop currently. Once I post it I'd really like to get some input to kind of speed up the process. I'll try to get script uploaded within the next 3 hours or so.
• 05-16-2012, 08:58 AM
Re: Removing patterns from Numeric wordlist
here is what I came up with. please let me know if it doesn't work I crudely cut and pasted the code from what will appear in my script:

Quote:

#!/bin/sh
function f_random () {
echo
echo "What is the minimum password string length you would like to use?"
echo
echo
echo "What is the maximum password string length you would like to use?"
echo
echo
echo "Enter the character set you would like to use."
echo
echo
echo "Enter the length of consecutive characters you would like to exclude."
echo "(e.x. 3)"
echo
echo
echo "What is the maximum number of one character you would like to use?"
echo "(e.x. 3)"
echo
echo
echo "What would you like the output file to be named?"
echo "(the file extension .lst will be appended to your selection)"
echo
echo
echo "Where would you like \$output.lst to be placed?"
echo "(e.x. /root/Desktop)"
echo
while [ ! -d "\$destination" ]
do
echo
echo "Directory cannot be found or does not exist"
echo
sleep 1
echo "Would you like to create a folder for the directory you selected? (y/n)"
if [ \$newdir = "y" ]; then
mkdir "\$destination"
while [ ! -d "\$destination" ]
do
echo "Folder: \$destination still cannot be found, starting over..."
sleep 2
f_random
done
else
sleep 1
echo "Where would you like the output word list to be placed?"
echo "(e.x. /root/Desktop/)"
fi
done
echo
echo "Would you like to go ahead and create \$output.lst in \$destination? (y/n)"
echo
echo
if [ "\$create" = "y" ]; then
echo
let consec=\$((\$consec - 1))
do
filter=\$(echo \$line | sed "/\([^A-Za-z0-9_]\|[A-Za-z0-9]\)\1\{\$consec,\}/d" | sed "/^\$/d")
if [ "\$filter" != "" ]; then
stopp="no"
number=1
filter3=\$(echo \$filter | fold -w1 | sort | uniq -c)
count=\$(echo \$line | wc -c)
let count=\$((\$count - 1))
until [[ "\$number" -gt "\$count" || "\$stopp" == "yes" ]]
do
filter4=\$(echo \$filter3 | awk '{print \$NF ":" \$'\$number'}' | sed -n 's/[^:]*://p')
let number=\$((\$number + 2))
if [ "\$filter4" -gt "\$maxchar" ]; then
stopp="yes"
else
stopp="no"
fi
done
if [ "\$stopp" == "no" ] && [ "\$number" -gt "\$count" ] ; then
echo "Writing: \$filter"
echo \$filter | cat >> "\$destination"/"\$output".lst
else
echo
echo "Skipping string: \$filter"
fi
else
echo
fi
done
if [ -e "\$destination"/"\$output".lst ]; then
echo
echo "\$output.lst exists in \$destination"
sed -i -e '1d' "\$destination"/"\$output".lst
sed -i -e '1d' "\$destination"/"\$output".lst
sed -i -e '1d' "\$destination"/"\$output".lst
sed -i -e '1d' "\$destination"/"\$output".lst
echo
sleep 2
echo "Successfully created \$output.lst"
echo
sleep 2
echo "Returning to the main menu..."
echo
sleep 2
else
echo
echo "\$output.lst does not exist in \$destination"
echo
sleep 2
echo "Unsuccessfully created \$output.lst"
echo
sleep 2
echo "Starting over..."
echo
sleep 2
f_random
fi
elif [ "\$create" = "n" ]; then
echo
else
echo
fi
}
f_random
• 06-17-2012, 09:49 AM
ternarybit
Re: Removing patterns from Numeric wordlist
Quote:

Originally Posted by zombie22
Numeric, and otherwise, wordlists have patterns that are unlikely to be utilized by software which creates the key based on 'random' digits. Repetition is an obvious one, but all patterns must go.

0000000001
0101010101
89abcdef
febf2cff (utilizing the 'f' 4/8 makes this key highly improbable to be generated randomly)

Obviously, this can drastically reduce a wordlist's size; a necessity if you want to challenge larger keys.

It seems this has been covered elsewhere but I cannot find a reference: Does anyone know of an existing program that does such a thing, or am I delusional? I realize there are numerous rudimentary ways to do this, but its seems as if there is already a program specifically for this. If not, its off to the coding board!

Your first premise is incorrect. Randomness requires that all possibilities are equally likely to occur, even "simple" ones.

Your second premise is also incorrect. Removing basic patterns from a comprehensive keyspace wordlist does not "drastically reduce a wordlist's size," depending somewhat on how you define "drastically." The patterns in your example occur very infrequently in a randomized key space. Try running crunch with the -d flag to limit repetitions and see how little it affects the resulting wordlist.

Consider this trade-off: will the computing time saved using a truncated wordlist really be less than the time spent generating said wordlist? Is it also worth the risk that you may truncate the target passphrase?

Unless you know with absolute certainty that the target passphrase was generated using specialized pattern-avoidance methods, I suggest you keep your dictionary complete.
Show 40 post(s) from this thread on one page
Page 2 of 2 First 12