Free Online Robots.Txt Generator

Free Online Robots.Txt Generator

For many amateur and startup webmasters configuring robots.txt is a little tedious task. Robots.txt is a simple plain vanilla text file in the root directory of a website to allow/deny web crawlers to particular locations of the website.

For a complete comprehensive discussion of robots.txt, check out Web Robots.txt here. I just came across a free tool which supports online generation of robots.txt

Webservice: Yellowpipe Robots Txt Generator

Features:

1. Free
2. Online hosted webservice
3. Supports 'Deny All'
4. Supports Configuration of Specific Folders
5. Supports Configuration of Compiled List of Robots