Methabot

  897 downloads
1.6.0.1 ISC License (ISCL)    
5.0/5 2
File system crawler

description

download

specifications

changelog

The Methabot software is a speed-optimized, scriptable and highly configurable web, ftp and local file system crawler. It supports scripted filetype parsing, a wide variety of customization options and is easily configured to fit anyones particular needs.

With the use of the module system and scripting language, users are able to take full or partial control of the crawling process and decide however Methabot should store web data, statistics and much more.

Just by running Methabot from command line you are able configure custom filetypes, filtering expressions, behaviour, and much more, so you don't have to be a scripter!
read more   
Last updated on February 23rd, 2009
1  
Methabot - Usage message

0 User reviews so far.

SUBMIT