robots.txt 1.0.3

robots.txt is a PHP script that acts like a normal robots.txt file, but with a few differences.

  Add it to your Download Basket!

 Add it to your Watch List!

0/5

Rate it!
send us
an update
LICENSE TYPE:
GPL (GNU General Public License) 
USER RATING:
UNRATED
  0.0/5
DEVELOPED BY:
Andrew Smith
HOMEPAGE:
somesound.org
CATEGORY:
ROOT \ Internet \ HTTP (WWW)
robots.txt is a script in PHP that acts like a normal robots.txt file, but with a few differences.

When a Spider attempts to access robots.txt, the script will "disallow" access to a list of pre-defined directories. When a normal user attempts to access robots.txt, it will "disallow" access to all directories.

This is a security precuation and an attempt to prevent users from finding "hidden" directories by looking at the robots.txt file.

What's New in This Release:

The two serious bugs that meant robots.txt did not function properly were fixed.
A bug that meant the data printed back would not be parseable by Web crawlers was fixed.
A new function to clean up data from the files was added.

Last updated on March 1st, 2006

#PHP script #site robots #web spider #robots #site #PHP #script

Add your review!

SUBMIT