Googlebot Question

"Terry" wondered:
: I wonder how Googlebot decides which sites to crawl and if there is a
: way to invite Googlebot.

Here's a way that I just heard about. View the site in question in the free
version of Opera (with the ads at the top). Google will go to that site
afterwards.

I found out about this because I gave a webmaster a link to a temporary
page, in a directory that's blocked from search engines, and 2 minutes after
that person visited that page, Google paid a visit too. When I asked him if
Google had followed him, he said yes, and explained that he uses the free
version of Opera rather than the paid one because Google follows links in
the address bar of the free version of Opera = a faster way to get pages
indexed.

Lois
 >> Stay informed about: Googlebot Question 

Terry wrote:

 > I have 4 sites all hosted by Yahoo. From the website logs, I see that 2
 > of the sites get visited by Googlebot every few days, but the other 2
 > sites haven't been visited by Googlebot during the past 2 months. Is
 > this normal?

Maybe... I have no idea about what kind of sites you are
talking, what they're about, how they're optimised, etc.

 > I wonder how Googlebot decides which sites to crawl

<a style='text-decoration: underline;' href="http://www.google.com/webmasters/1.html#B2" target="_blank">http://www.google.com/webmasters/1.html#B2</a>

 > and if there is a way to invite Googlebot.

Have a spidered site link to them.

 > I did the "Add URL" on Google, but that didn't
 > seem to work.

Just patience maybe?

--
Els
<a style='text-decoration: underline;' href="http://locusmeus.com/" target="_blank">http://locusmeus.com/</a>
Sonhos vem. Sonhos vão. O resto é imperfeito.
- Renato Russo -<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Googlebot Question 

Lois said what up foo and said:
 > "Terry" wondered:
  >> I wonder how Googlebot decides which sites to crawl and if there is a
  >> way to invite Googlebot.
 >
 > Here's a way that I just heard about. View the site in question in
 > the free version of Opera (with the ads at the top). Google will go
 > to that site afterwards.
 >
 > I found out about this because I gave a webmaster a link to a
 > temporary page, in a directory that's blocked from search engines,
 > and 2 minutes after that person visited that page, Google paid a
 > visit too. When I asked him if Google had followed him, he said yes,
 > and explained that he uses the free version of Opera rather than the
 > paid one because Google follows links in the address bar of the free
 > version of Opera = a faster way to get pages indexed.
 >
 > Lois

Can someone yell, Spyware!? j/k...

--
Sharif T. Karim
....you expected some spam here?...<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Googlebot Question 

Lois wrote:

 > "Terry" wondered:
 > : I wonder how Googlebot decides which sites to crawl and if there is a
 > : way to invite Googlebot.
 >
 > Here's a way that I just heard about. View the site in question in the free
 > version of Opera (with the ads at the top). Google will go to that site
 > afterwards.
 >
 > I found out about this because I gave a webmaster a link to a temporary
 > page, in a directory that's blocked from search engines, and 2 minutes after
 > that person visited that page, Google paid a visit too. When I asked him if
 > Google had followed him, he said yes, and explained that he uses the free
 > version of Opera rather than the paid one because Google follows links in
 > the address bar of the free version of Opera = a faster way to get pages
 > indexed.

The page is searched, in order to pick the right advert to
go with the viewed page. Only sites that are indexed by
Google's Mediabot however, get to see a different ad than
"Buy Opera now".
I just checked myself, as I have exactly one page indexed by
the Mediabot. The ad that it gave me, had nothing to do with
the page though.
The page is <a style='text-decoration: underline;' href="http://locusmeus.com/html-css/pageslipping.html," target="_blank">http://locusmeus.com/html-css/pageslipping.html,</a>
and the ad is the Dutch translation of: "Get slimmer NOW!
with apple vinegar 100% natural."

Just out of curiosity, wondering how they think a CSS method
to save an HTML page has anything to do with apple
vinegar... Could any of you free Opera users check what ad
they get with that page?

--
Els
<a style='text-decoration: underline;' href="http://locusmeus.com/" target="_blank">http://locusmeus.com/</a>
Sonhos vem. Sonhos vão. O resto é imperfeito.
- Renato Russo -<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Googlebot Question 

"Els" wrote:
: The page is searched, in order to pick the right advert to
: go with the viewed page.

The page doesn't need to be searched for Google to follow the link, though.


Only sites that are indexed by
: Google's Mediabot however, get to see a different ad than
: "Buy Opera now".
: I just checked myself, as I have exactly one page indexed by
: the Mediabot. The ad that it gave me, had nothing to do with
: the page though.
: The page is http://locusmeus.com/html-css/pageslipping.html,
: and the ad is the Dutch translation of: "Get slimmer NOW!
: with apple vinegar 100% natural."
:
: Just out of curiosity, wondering how they think a CSS method
: to save an HTML page has anything to do with apple
: vinegar... Could any of you free Opera users check what ad
: they get with that page?

The ad has no connection for me either. It's a Father's Day ad, and it
appeared before your page appeared. The same ad was there when I went to
other sites.

Lois
 >> Stay informed about: Googlebot Question 

Lois wrote:

 > "Els" wrote:
 > : The page is searched, in order to pick the right advert to
 > : go with the viewed page.
 >
 > The page doesn't need to be searched for Google to follow the link, though.

So you're saying that if I take any orphan page and display
it in Opera, Google will find it.
I'll check that in a minute, and see how long it takes for
Google to come and see it

 > Only sites that are indexed by
 > : Google's Mediabot however, get to see a different ad than
 > : "Buy Opera now".
 > : I just checked myself, as I have exactly one page indexed by
 > : the Mediabot. The ad that it gave me, had nothing to do with
 > : the page though.
<font color=purple> > : The page is <a style='text-decoration: underline;' href="http://locusmeus.com/html-css/pageslipping.html,</font" target="_blank">http://locusmeus.com/html-css/pageslipping.html,</font</a>>
 > : and the ad is the Dutch translation of: "Get slimmer NOW!
 > : with apple vinegar 100% natural."
 > :
 > : Just out of curiosity, wondering how they think a CSS method
 > : to save an HTML page has anything to do with apple
 > : vinegar... Could any of you free Opera users check what ad
 > : they get with that page?
 >
 > The ad has no connection for me either. It's a Father's Day ad, and it
 > appeared before your page appeared. The same ad was there when I went to
 > other sites.

That doesn't make the story very logical. If Google really
is working with Opera to get the right ads in there, they
could do better than that, I suppose?

--
Els
<a style='text-decoration: underline;' href="http://locusmeus.com/" target="_blank">http://locusmeus.com/</a>
Sonhos vem. Sonhos vão. O resto é imperfeito.
- Renato Russo -<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Googlebot Question 

"Els" wrote:
: So you're saying that if I take any orphan page and display
: it in Opera, Google will find it.

That's what I was told, and the page that Google found was an orphan page,
online for only a couple of weeks.


: I'll check that in a minute, and see how long it takes for
: Google to come and see it

It was 2 minutes for me.

Lois
 >> Stay informed about: Googlebot Question 

Els wrote:

 > That doesn't make the story very logical. If Google really is working
 > with Opera to get the right ads in there, they could do better than
 > that, I suppose?

If no one bought keywords on your page a place holder add is used, I guess.

--
John MexIT: <a style='text-decoration: underline;' href="http://johnbokma.com/mexit/" target="_blank">http://johnbokma.com/mexit/</a>
personal page: <a style='text-decoration: underline;' href="http://johnbokma.com/" target="_blank">http://johnbokma.com/</a>
Experienced Perl programmer available: <a style='text-decoration: underline;' href="http://castleamber.com/" target="_blank">http://castleamber.com/</a>
Happy Customers: <a style='text-decoration: underline;' href="http://castleamber.com/testimonials.html" target="_blank">http://castleamber.com/testimonials.html</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Googlebot Question 

Lois wrote:

 > "Els" wrote:
 > : So you're saying that if I take any orphan page and display
 > : it in Opera, Google will find it.
 >
 > That's what I was told, and the page that Google found was an orphan page,
 > online for only a couple of weeks.
 >
 > : I'll check that in a minute, and see how long it takes for
 > : Google to come and see it
 >
 > It was 2 minutes for me.

Did your page appear in the search results, or did you just see a bot?
Notice that sometimes orphan pages get out. I once posted a link to a
picture, and somehow the page above it ended up in Google. Someone
probably did some URL hacking, and linked to me.

--
John MexIT: <a style='text-decoration: underline;' href="http://johnbokma.com/mexit/" target="_blank">http://johnbokma.com/mexit/</a>
personal page: <a style='text-decoration: underline;' href="http://johnbokma.com/" target="_blank">http://johnbokma.com/</a>
Experienced Perl programmer available: <a style='text-decoration: underline;' href="http://castleamber.com/" target="_blank">http://castleamber.com/</a>
Happy Customers: <a style='text-decoration: underline;' href="http://castleamber.com/testimonials.html" target="_blank">http://castleamber.com/testimonials.html</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Googlebot Question 

"John Bokma" <postmaster.RemoveThis@castleamber.com> wrote in message
news:40c7986c$0$208$58c7af7e@news.kabelfoon.nl...

 > If no one bought keywords on your page a place holder add is used, I
guess.

Usually charity ones though? I think Els' was for a slimming product.
--
Charles Sweeney
<a style='text-decoration: underline;' href="http://www.CharlesSweeney.com" target="_blank">www.CharlesSweeney.com</a><!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Googlebot Question 

Lois wrote:

 > "Els" wrote:
 > : So you're saying that if I take any orphan page and display
 > : it in Opera, Google will find it.
 >
 > That's what I was told, and the page that Google found was an orphan page,
 > online for only a couple of weeks.
 >
 > : I'll check that in a minute, and see how long it takes for
 > : Google to come and see it
 >
 > It was 2 minutes for me.

Well, it's been 12 minutes now that I've been viewing the
page in Opera, but no bot yet. I'll check tomorrow though

--
Els
<a style='text-decoration: underline;' href="http://locusmeus.com/" target="_blank">http://locusmeus.com/</a>
Sonhos vem. Sonhos vão. O resto é imperfeito.
- Renato Russo -<!-- ~MESSAGE_AFTER~ -->
 >> Stay informed about: Googlebot Question 

In article <2ipgh2Fpfun7U27.TakeThisOut@uni-berlin.de>,
Els <els.aNOSPAM.TakeThisOut@tiscali.nl> wrote:
} Lois wrote:
} > Here's a way that I just heard about. View the site in question in the free
} > version of Opera (with the ads at the top). Google will go to that site
} > afterwards.
}
} The page is searched, in order to pick the right advert to
} go with the viewed page. Only sites that are indexed by
} Google's Mediabot however, get to see a different ad than
} "Buy Opera now".
} I just checked myself, as I have exactly one page indexed by
} the Mediabot. The ad that it gave me, had nothing to do with

You're right, Els.

I recently discussed this with Google's tech support. The "Mediapartners-Google"
bot and the "real" Google bot are independent. The former will indeed
visit any site using Google ads, but only to index the page for ad
serving.

peace,

-*-
Charles M. Kozierok (mailto:ixlu@PCGuide.com)
Author, The TCP/IP Guide - <http://www.TCPIPGuide.com>
A comprehensive, comprehensible guide to TCP/IP protocols and technologies...
*NEW* Version 2.0 released -- Full Guide now online in HTML for FREE!
 >> Stay informed about: Googlebot Question