Installing CeWL in Backtrack4

[skidmarq]

A great way to build custom password lists to feed into password crackers is by profiling the target’s websites using CeWL. More information on CeWL can be found here: CeWL - DigiNinja

Getting CeWL installed on BT4 takes a little bit of work. Since I just got done doing this very thing, i figured I'd share the steps needed to do the trick. The first thing to do is download the latest version of Ruby Gems (BT4 comes with 1.2.0, I believe).

wget http://rubyforge.org/frs/download.ph...gems-1.3.6.tgz
tar -xvf rubygems-1.3.6.tgz
rm rubygems-1.3.6.tgz
cd rubygems-1.3.6/
ruby setup.rb
gem –v (verifying the version is 1.3.6)

Once this is complete, download the latest version of CeWL from the project's website.

cd /pentest/passwords
wget digininja.org/files/cewl_3.0.tar.bz2
tar –xvjf cewl_3.0.tar.bz2
rm cewl_3.0.tar.bz2
cd cewl

Now there are some dependencies needed to run the program.

apt-get install libxml2-dev libxslt-dev libimage-exiftool-perl
gem install mime-types archive-tar-minitar nokogiri echoe hoe rcov 
gem install rubyzip mini_exiftool http_configuration spider hpricot
export RUBYOPT=rubygems

Once we’re at this point, test it out and make sure it is functional.

./cewl.rb -d 2 -v <website of choice>

Assuming it works you can now begin creating custom password lists based on our target of choice.

[psychot3ch1]

Great Tutorial dude! very informative and very complete thank you for the tutorial keep up the good work!

cheers!

[FluxCapacitor]

Getting:

wompus@bt:/pentest/passwords/cewl$ ./cewl.rb
./cewl.rb:58:in `require': no such file to load -- spider (LoadError)
        from ./cewl.rb:58

Any idea why this is happening in case anyone else has the same issue?

[skidmarq]

Hi FluxCapacitor, it looks like it might be a problem loading the appropriate gem package (spider).

When you performed the "gem install mime-types archive-tar-minitar nokogiri echoe hoe rcov zip rubyzip mini_exiftool http_configuration spider hpricot" command, did it give you any errors?

[davehardy20]

Hi FluxCapacitor, I too get the same error as skidmarq, I dont get any errors when installing the GEMS packages but do get an error when it tries to install zip reports ERROR: could not find gem zip locally or in a repository.

[skidmarq]

I do not believe zip is an absolute requirement to run the application...are you able to subsequently run the app even with the error?

[davehardy20]

No I cannot run CeWL I get the same error as FluxCapacitor gets

./cewl.rb:58:in `require': no such file to load -- spider (LoadError)
from ./cewl.rb:58

I just followed the instructions you gave.

[skidmarq]

Guys, try to re-run the command without zip and see if that does the trick (even possibly one at a time).

"gem install mime-types archive-tar-minitar nokogiri echoe hoe rcov rubyzip mini_exiftool http_configuration spider hpricot"

If that doesn't work then I'll install again on my other box to see if I can get to the bottom of it...

[davehardy20]

Hi Skidmarq,

Below is the output from the gem install

root@bt:~# gem install mime-types archive-tar-minitar nokogiri echoe hoe rcov rubyzip mini_exiftool http_configuration spider hpricot
Successfully installed mime-types-1.16
Successfully installed archive-tar-minitar-0.5.2
Building native extensions. This could take a while...
Successfully installed nokogiri-1.4.1
Successfully installed echoe-4.3.1
Successfully installed hoe-2.6.0
Building native extensions. This could take a while...
Successfully installed rcov-0.9.8
Successfully installed rubyzip-0.9.4
Successfully installed mini_exiftool-1.0.1
Successfully installed http_configuration-1.0.2
Successfully installed spider-0.4.4
Building native extensions. This could take a while...
Successfully installed hpricot-0.8.2
11 gems installed
Installing ri documentation for mime-types-1.16...
Installing ri documentation for archive-tar-minitar-0.5.2...
Installing ri documentation for nokogiri-1.4.1...

No definition for parse_memory

No definition for parse_file

No definition for parse_with

No definition for get_options

No definition for set_options
Installing ri documentation for echoe-4.3.1...
Installing ri documentation for hoe-2.6.0...
Installing ri documentation for rcov-0.9.8...
Installing ri documentation for rubyzip-0.9.4...
Installing ri documentation for mini_exiftool-1.0.1...
Installing ri documentation for http_configuration-1.0.2...
Installing ri documentation for spider-0.4.4...
Installing ri documentation for hpricot-0.8.2...
Installing RDoc documentation for mime-types-1.16...
Installing RDoc documentation for archive-tar-minitar-0.5.2...
Installing RDoc documentation for nokogiri-1.4.1...

No definition for parse_memory

No definition for parse_file

No definition for parse_with

No definition for get_options

No definition for set_options
Installing RDoc documentation for echoe-4.3.1...
Installing RDoc documentation for hoe-2.6.0...
Installing RDoc documentation for rcov-0.9.8...
Installing RDoc documentation for rubyzip-0.9.4...
Installing RDoc documentation for mini_exiftool-1.0.1...
Installing RDoc documentation for http_configuration-1.0.2...
Installing RDoc documentation for spider-0.4.4...
Installing RDoc documentation for hpricot-0.8.2...
root@bt:~#

CeWL still does not run, still get error

./cewl.rb:58:in `require': no such file to load -- spider (LoadError)
from ./cewl.rb:58

[skidmarq]

Can you provide me with the output from the "gem list" command?

Thanks!

Also, make sure to re-run "export RUBYOPT=rubygems" in case that didn't complete during the initial install due to the gem install failing...

I just followed the instructions off a brand new install of BT4 and it worked. The only change I made was removing "zip" from the list of installed ruby extensions. I've edited my original post as I fear that as soon as it tries to install zip, it might brainfart the rest of the process.

As mentioned earlier, simply try to reset your environmental path using the "export RUBYOPT=rubygems" command. I believe this should do the trick...