Posted by : Deepak Vasudevan Wednesday, October 08, 2008

Free Online Robots.Txt Generator

For many amateur and startup webmasters configuring robots.txt is a little tedious task. Robots.txt is a simple plain vanilla text file in the root directory of a website to allow/deny web crawlers to particular locations of the website.

For a complete comprehensive discussion of robots.txt, check out Web Robots.txt here. I just came across a free tool which supports online generation of robots.txt

Webservice: Yellowpipe Robots Txt Generator


  1. Free
  2. Online hosted webservice
  3. Supports 'Deny All'
  4. Supports Configuration of Specific Folders
  5. Supports Configuration of Compiled List of Robots

Popular Post

Blogger templates


Blog Archive


Blog Archive

Powered by Blogger.

- Copyright © Lavanya Deepak -Metrominimalist- Powered by Blogger - Designed by Johanes Djogan -