As you could see in the code I pasted in the previous post about Meta tags there's a part that says:
meta name="robots" content="..., ..."
meta name="googlebot" content="..., ..."
meta name="msnbot" content="..., ..."
These meta tags can control the behavior of search engine crawling and indexing. The robots meta tag applies to all search engines, while the "googlebot" meta tag is specific to Google. The same with msnbot, which doesn't allow MSN to crawl your site.
The default values are "index, follow" (the same as "all") and do not need to be specified. However, when you want to specify something to the crawler, you have to use something like (when specifying multiple values, separate them with a comma):
* noindex: prevents the page from being indexed.
* nofollow: prevents the crawler from following links from this page. This means that Google/Bing does not transfer PageRank or anchor text across these links.
* noodp: prevents the alternative description from the ODP/DMOZ from being used as your site description. As you know, Google/Bing use several sources to describe your site in the Search Results. One of them is DMOZ. If you don't want them to use that you'll have to use this combinations:
If you only want to avoid Google to use it, you only have to use googlebot in the name: meta name=googlebot.
In case of MSNbot:meta name=msnbot
* noarchive: prevents Google/Bing from showing the Cached link for a page.
* unavailable_after:[date]: lets you specify the exact time and date you want to stop crawling and indexing of this page.
* nosnippet: prevents a snippet from being shown in the search results
0 comments
Post a Comment