I managed to fix this by passing in the -n option (for ex "-n 1000")
I also noticed it pulls links from google with some extra params that need to be stripped out. It seems google changed how they link to files like PDFs and DOC.
For example, it has
/url?q=http://YOURDOMAIN.com/document.pdf&sa=U&ei=7asadsddsapUK_uDo3Qqw G19YGgAQ&ved=309BgQFjAC&usg=AFQjCNH2OiklnA vR8Zp11ESHs0KzMsd9Mg
So in downloader.py I modified the "downloader.py" file, line 4 from:
to
Code:
self.url=url.replace("/url?q=", "", 1).split("&")[0]