Methabot 1.6.0.1

File system crawler

  Add it to your Download Basket!

 Add it to your Watch List!

0/5

Rate it!

What's new in Methabot 1.6.0.1:

  • Bugfix, when external-peek was used the depth limit was messed up.
  • Memory usage cleanup fixes
  • dynamic-url option is no longer set to lookup by default, since it slows down the crawling significantly
  • Build system now creates and installs some header files that modules can use when linking
Read full changelog
send us
an update
LICENSE TYPE:
ISC License (ISCL) 
USER RATING:
5.0/5 2
DEVELOPED BY:
Emil Romanus
HOMEPAGE:
bithack.se
CATEGORY:
ROOT \ Internet \ FTP
1 Methabot Screenshot:
Methabot - Usage message
The Methabot software is a speed-optimized, scriptable and highly configurable web, ftp and local file system crawler. It supports scripted filetype parsing, a wide variety of customization options and is easily configured to fit anyones particular needs.

With the use of the module system and scripting language, users are able to take full or partial control of the crawling process and decide however Methabot should store web data, statistics and much more.

Just by running Methabot from command line you are able configure custom filetypes, filtering expressions, behaviour, and much more, so you don't have to be a scripter!

Last updated on February 23rd, 2009

feature list requirements

#web crawler #ftp crawler #filesystem crawler #web #ftp #crawler #bot

Add your review!

SUBMIT