Where should I place robots.txt

Need help? Post your questions here.

Moderator: moderators

Where should I place robots.txt

Postby mapes » Wed Aug 06, 2014 7:48 pm

If I wanted to deny something from trying to index my subsonic server where should I place the robots.txt?
mapes
 
Posts: 11
Joined: Wed Apr 29, 2009 4:44 pm

Re: Where should I place robots.txt

Postby GJ51 » Wed Aug 06, 2014 9:30 pm

Where to put it

The short answer: in the top-level directory of your web server.

The longer answer:

When a robot looks for the "/robots.txt" file for URL, it strips the path component from the URL (everything from the first single slash), and puts "/robots.txt" in its place.

For example, for "http://www.example.com/shop/index.html, it will remove the "/shop/index.html", and replace it with "/robots.txt", and will end up with "http://www.example.com/robots.txt".

So, as a web site owner you need to put it in the right place on your web server for that resulting URL to work. Usually that is the same place where you put your web site's main "index.html" welcome page. Where exactly that is, and how to put the file there, depends on your web server software.

Remember to use all lower case for the filename: "robots.txt", not "Robots.TXT.

See also:
◦What program should I use to create /robots.txt?
◦How do I use /robots.txt on a virtual host?
◦How do I use /robots.txt on a shared host?

http://www.robotstxt.org/robotstxt.html
Gary J

http://bios-mods.com
http://www.maplegrovepartners.com
http://theaverageguy.tv/category/tagpodcasts/cyberfrontiers/
User avatar
GJ51
 
Posts: 3492
Joined: Wed Oct 20, 2010 11:58 pm
Location: Western New York


Return to Help

Who is online

Users browsing this forum: No registered users and 32 guests