| View previous topic :: View next topic |
| Author |
Message |
comprookie2000
Retired Dev
Joined: 25 Jul 2004
Posts: 925
Location: Sun City Center, Florida
|
 Posted: Thu Jul 14, 2005 1:37 am Post subject: googlebot running scripts? |
 |
|
What does this mean?
| Code: | [Wed Jul 13 00:26:17 2005] [error] [client 207.46.98.118] File does not exist: /var/www/localhost/htdocs/robots.txt
[client 66.249.65.73] script '/var/www/localhost/htdocs/phpBB2/viewforum.php' not found or unable to stat
[Wed Jul 13 00:58:15 2005] [error] [client 66.249.64.14] File does not exist: /var/www/localhost/htdocs/robots.txt
[Wed Jul 13 09:25:48 2005] [error] [client 207.68.146.49] File does not exist: /var/www/localhost/htdocs/robots.txt
[Wed Jul 13 12:48:12 2005] [error] [client 207.68.146.49] File does not exist: /var/www/localhost/htdocs/robots.txt |
to see who it was I ran nmap | Code: |
david@abbottdavid ~ $ nmap -sT -p 21,22,23,25,80,110,135-139,143,443,445,5000 -T 3 66.249.65.73
Starting nmap 3.81 ( http://www.insecure.org/nmap/ ) at 2005-07-13 21:30 EDT
Note: Host seems down. If it is really up, but blocking our ping probes, try -P0
Nmap finished: 1 IP address (0 hosts up) scanned in 4.000 seconds
david@abbottdavid ~ $ nmap -P0 -p 21,22,23,25,80,110,135-139,143,443,445,5000 -T 3 66.249.65.73
Starting nmap 3.81 ( http://www.insecure.org/nmap/ ) at 2005-07-13 21:31 EDT
All 15 scanned ports on crawl-66-249-65-73.googlebot.com (66.249.65.73) are: filtered |
Nmap finished: 1 IP address (1 host up) scanned in 16.471 seconds
david@abbottdavid ~ $
_________________
http://dev.gentoo.org/~dabbott/ |
|
| Back to top |
|
 |
bruciemoose
n00b
Joined: 22 Sep 2004
Posts: 19
Location: Llanfair TH, North Wales
|
| Posted: Thu Jul 14, 2005 2:44 am Post subject: Use resolveip instead. |
|
|
nmap can get you into trouble. Much safer to simply do a reverse DNS lookup with resolveip:
| Code: |
sven@brucie ~ $ resolveip 207.46.98.118
Host name of 207.46.98.118 is msnbot.msn.com
sven@brucie ~ $ resolveip 66.249.64.14
Host name of 66.249.64.14 is crawl-66-249-64-14.googlebot.com
|
Is that all you were interested in? The 404s for robots.txt are normal - most crawlers look for a robots.txt, which is a file to restrict the indexing of certain pages / directories of your site.
Andrew
_________________
Teachers: Write student reports online! |
|
| Back to top |
|
|
comprookie2000
Retired Dev
Joined: 25 Jul 2004
Posts: 925
Location: Sun City Center, Florida
|
| Posted: Thu Jul 14, 2005 8:48 am Post subject: |
|
|
This is what I was looking at;
[client 66.249.65.73] script '/var/www/localhost/htdocs/phpBB2/viewforum.php' not found or unable to stat
?
_________________
http://dev.gentoo.org/~dabbott/ |
|
| Back to top |
|
|
rex123
Apprentice
Joined: 21 Apr 2004
Posts: 272
|
| Posted: Thu Jul 14, 2005 9:03 am Post subject: |
|
|
Maybe you used to have phpBB installed? All that error is telling you is that someone tried to make an http request to a resource that isn't there.
If the resource is a normal file, the error is "File does not exist". If it's cgi you get "script X not found or unable to stat"
This is from my logs: an example of the 2 errors from a boring automated bot that's trying to hack random awstats installations:
| Code: | [Tue Jul 12 08:05:47 2005] [error] [client 217.160.255.9] File does not exist: /var/www/localhost/htdocs/awstats.pl
[Tue Jul 12 08:05:47 2005] [error] [client 217.160.255.9] script not found or unable to stat: /var/www/localhost/cgi-bin/awstats.pl
|
Googlebot is only following links, so either you used to have phpBB installed, or someone somewhere has put a link to that URL. Either way, you shouldn't worry about it. |
|
| Back to top |
|
|
trossachs
Veteran
Joined: 22 Jan 2004
Posts: 1204
Location: London
|
| Posted: Sat Aug 06, 2005 4:33 pm Post subject: |
|
|
| Do you use Google AdSense or AdWords? I do, GGbot simply looks up websites to check advertising. It also checks broken links. As rex123 says, nothing to worry about. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
