Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
robots.txt?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
handsomepete
Guru
Guru


Joined: 21 Apr 2002
Posts: 548
Location: Kansas City, MO

PostPosted: Wed May 29, 2002 5:17 am    Post subject: robots.txt? Reply with quote

I've got this file (robots.txt) in my /usr/portage directory. Google tells me that it's supposed to be something that search engines source to see what they can and can't catalog (or something like that), but I don't know how it would've gotten in the portage directory. I'm new to this webserver stuff, so any ideas/opinions would be appreciated. According to Apache's logs various IPs try to access it every now and again.

Code:

:/usr/portage:
[jeff@lydia.nerd.obsessive] >cat robots.txt
User-agent: *
Disallow: /
Back to top
View user's profile Send private message
fghellar
Bodhisattva
Bodhisattva


Joined: 10 Apr 2002
Posts: 856
Location: Porto Alegre, BR

PostPosted: Wed May 29, 2002 5:23 am    Post subject: Re: robots.txt? Reply with quote

handsomepete wrote:
Google tells me that it's supposed to be something that search engines source to see what they can and can't catalog (or something like that)

Wget also respects it, so whenever you run wget on a directory (not any specific file), it will try to find this file first.
_________________
| www.gentoo.org | www.tldp.org | www.google.com |
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum