Free Online Robots.Txt Generator
For many amateur and startup webmasters configuring robots.txt is a little tedious task. Robots.txt is a simple plain vanilla text file in the root directory of a website to allow/deny web crawlers to particular locations of the website.
For a complete comprehensive discussion of robots.txt, check out Web Robots.txt here. I just came across a free tool which supports online generation of robots.txt
Webservice: Yellowpipe Robots Txt Generator
Features:
For many amateur and startup webmasters configuring robots.txt is a little tedious task. Robots.txt is a simple plain vanilla text file in the root directory of a website to allow/deny web crawlers to particular locations of the website.
For a complete comprehensive discussion of robots.txt, check out Web Robots.txt here. I just came across a free tool which supports online generation of robots.txt
Webservice: Yellowpipe Robots Txt Generator
Features:
- Free
- Online hosted webservice
- Supports 'Deny All'
- Supports Configuration of Specific Folders
- Supports Configuration of Compiled List of Robots
No comments:
Post a Comment