• "Thank you for the time you have spent helping me. It is genuinely appreciated."
• "Thank you to everyone who helped with my problem! My creaky old 1996 era website is all cleaned up and doing just fine now! Good Guys Rule!"
• "WOW!!!!!!!! Thank you so much for your help! I was reluctant to post because I thought you guys might think my site is too small, too insignificant, etc. Thanks so much! To me, it's a BIG deal!"
• "My traffic has doubled and I am now either top or close to the top in search terms"
• "Thanks. Hopefully my late night paid off then, all the help and information has been great!!!!!!!"
• "Wow Webado Thank You Again.. You Really Know Your stuff! (…) You Are a True professional and Seriously The Only Person That Could Even Figure This out. I even Spoke to Other Top Specialists and YOU were the Only one who told me what to do and what was wrong."
• "You are AMAZING! Thank you so much!"
• "Finally, thanks so much for your concern and prompt reply, Squibble. Can't imagine a person would deligated his efforts to someones you don't even know and met. (…) Thanks for your help!! It means everything to me!"
• "Wow thanks so much! I have just been to the hot springs in Banos Ecuador, the volcano is rumbling and the town has evacuated but I am still here talking Apache, now that is dedication, I will test your helpfulness out 2moro! Thanks so much!!!"
• "THANKYOU THANKYOU THANKYOU! YOU ARE SOOOO AMAZING! MY BLOG IS GONE FROM GOOGLE AND ALL THE OLD POSTS ASSOCIATED WITH THAT TOO! THANK YOU! I LOVE YOU!"
• "It worked! Thank you very much for your help Cristina :)"
• "I could NEVER have seen what's possible without this forum. I am so grateful."
• "Thank you so much for your detailed response to my questions. In 10 years of me having a website, no one has explained these concepts better to me than you did."
• "IT WORKED!!! Thank you so much for saving me the grief and embarrassment of this problem. I truly appreciate both your knowledge and guidance."
• "Thank you so much for such a detailed answer and putting into terms I can easily understand. (…) Where shall I send the batch of brownies?"
• "Thank you so much Squibble you are a hero. I have done what it says and i will check to see if i appear in google! Thanks again!"
• "you guys are amazing! thank you so much redleg! (…) and if you happen to ever be in san carlos give me a shout - you deserve at least a beer and a lunch! "
• "Thank you Squibble, Vanessa, Cristina, and Ishigaki for weighing in on this and helping me. I hope I can pay it forward one day."
• "Great info Guys. I really appreciate it. It was awesome of y'all to help me out. I really appreciate it. Thank you."
• "Your amazing :-) i love you lol xx im sorted now thanks and never in a million years would i have found that out!"
• "THANK YOU! I love Google and appreciate that they have these safety precautions in place for those nasty hackets - especially when we can fix the problem! Thanks so much for your help. Whew."
• "Thank you very much, Robbo! With a little tweaking it worked perfectly!"
• "yay!!! I think it finally went through - THANK YOU SO MUCH!!!"

Today Google introduced Instant Previews, a new search feature that helps people find information faster by showing a visual preview of each result. Traditionally, elements of the search results like the title, URL, and snippet—the text description in each result—help people determine which results are best for them. Instant Previews achieves the same goal with a visual representation of each page and where the relevant content is, instead of a text description. For our webmaster community, this presents an opportunity to reveal the design of your site and why your page is relevant for a particular query. We'd like to offer some thoughts on how to take advantage of the feature.

First of all, it's important to understand what the new feature does. When someone clicks on the magnifying glass on any result, a zoomed-out snapshot of the underlying page appears to the right of the results. Orange highlights indicate where highly relevant content on the page is, and text call outs show search terms in context.

These elements let people know what to expect if they click on that result, and why it's relevant for their query. Our testing shows that the feature really does help with picking the right result—using Instant Previews makes searchers 5% more likely to be satisfied with the results they click.

Many of you have put a lot of thought and effort into the structure of your sites, the layout of your pages, and the information you provide to visitors. Instant Previews gives people a glimpse into that design and indicates why your pages are relevant to their query. Here are some details about how to make good use of the feature.

• Keep your pages clearly laid out and structured, with a minimum of distractions or extraneous content. This is always good advice, since it improves the experience for visitors, and the simplicity and clarity of your site will be apparent via Instant Previews.
• Try to avoid interstitial pages, ad pop-ups, or other elements that interfere with your content. In some cases, these distracting elements may be picked up in the preview of your page, making the screenshots less attractive.
• Many pages have their previews generated as part of our regular crawl process. Occasionally, we will generate screenshots on the fly when a user needs it, and in these situations we will retrieve information from web pages using a new "Google Web Preview" user-agent.
• Instant Previews does not change our search algorithm or ranking in any way. It's the same results, in the same order. There is also no change to how clicks are tracked. If a user clicks on the title of a result and visits your site, it will count as a normal click, regardless of whether the result was previewed. Previewing a result, however, doesn't count as a click by itself.
• Currently, adding the nosnippet meta tag to your pages will cause them to not show a text snippet in our results. Since Instant Previews serves a similar purpose to snippets, pages with the nosnippet tag will also not show previews. However, we encourage you to think carefully about opting out of Instant Previews. Just like regular snippets, previews tend to be helpful to users—in our studies, results which were previewed were more than four times as likely to be clicked on. URLs that have been disallowed in the robots.txt file will also not show Instant Previews.
• Currently, some videos or Flash content in previews appear as a "puzzle piece" icon or a black square. We're working on rendering these rich content types accurately.

Posted by Jeremy Silber, Software Engineer

1. Wait for Googlebot to re-crawl and re-index the page
   This is the natural method for how most content is updated at Google. Sometimes it can take a fairly long time, depending on how frequently Googlebot currently crawls the page in question. Once we've re-crawled and re-indexed the page, the old content will usually not be visible as it'll be replaced by the current content. Provided Googlebot is not blocked from crawling the page in question (either by robots.txt or by not being able to access the server properly), you don't have to do anything special for this to take place. It's generally not possible to speed up crawling and indexing, as these processes are fully automated and depend on many external factors.
   
   
2. Use Google's public URL removal tool to request removal of content that has been removed from someone else's webpage
   Using this tool, it's necessary to enter the exact URL of the page that has been modified, select the "Content has been removed from the page" option, and then specify one or more words that have been completely removed from that page.
   
   
   
   Note that none of the words you enter can appear on the page; even if a word has been removed from one part of the page, your request will be denied if that word still appears on another part of the page. Be sure to choose a word (or words) that no longer appear anywhere on the page. If, in the above example, you removed "top secret velocitus incalculii capturing device," you should submit those words and not something like "my project." However, if the word "top" or "device" still exists anywhere on the page, the request would be denied. To maximize your chances of success, it's often easiest to just enter one word that you're sure no longer appears anywhere on the page.
   
   Once your request has been processed and it's found that the submitted word(s) no longer appear on the page, the search result will no longer show a snippet, nor will the cached page be available. The title and the URL of the page will still be visible, and the entry may still appear in search results for searches related to the content that has been removed (such as searches for [velocitus incalculii]), even if those words no longer appear in the snippet. However, once the page has been re-crawled and re-indexed, the new snippet and cached page can be visible in our search results.
   
   Keep in mind that we will need to verify removal of the word(s) by viewing the page. If the page no longer exists and the server is returning a proper 404 or 410 HTTP result code, making us unable to view the page, you may be better off requesting removal of the page altogether.
   
   
3. Use Google Webmaster Tools URL removal tool to request removal of information on a page from your website
   If you have access to the website in question and have verified ownership of it in Google Webmaster Tools, you can use the URL removal tool there (under Site Configuration > Crawler access) to request that the snippet and the cached page be removed until the page has been re-crawled. To use this tool, you only need to submit the exact URL of the page (you won't need to specify any removed words). Once your request has been processed, we'll remove the snippet and the cached page from search results. The title and the URL of the page will still be visible, and the page may also continue to rank in search results for queries related to content that has been removed. After the page has been re-crawled and re-indexed, the search result with an updated snippet and cached page (based on the new content) can be visible.

Google and language recognition

The anatomy of a multilingual site: URL structure

Crawling and indexing your multilingual website

Working with character encodings

If your tongue gets twisted...

Preparing for global websites

Planning multi-regional websites

• ccTLDs (country-code top level domain names): These are tied to a specific country (for example .de for Germany, .cn for China). Users and search engines use this as a strong sign that your website is explicitly for a certain country.
• gTLDs (generic top level domain names): These are not tied to a specific country. Examples of gTLds are .com, .net, .org, .museum. Google sees regional top level domain names such as .eu and .asia as gTLDs, since they cannot be tied to a specific country. We also treat some vanity ccTLDs (such as .tv, .me, etc.) as gTLDs as we've found that users and webmasters frequently see these as being more generic than country-targeted (we don't have a complete list of such vanity ccTLDs that we treat as gTLDs as it may change over time). You can set geotargeting for websites with gTLDs using the Webmaster Tools Geographic Target setting.

Geotargeting factors

1. Use of a ccTLD is generally a strong signal for users since it explicitly specifies a single country in an unmistakable way.
   or
   Webmaster Tools' manual geotargeting for gTLDs (this can be on a domain, subdomain or subdirectory level); more information on this can be found in our blog post and in the Help Center. With region tags from geotargeting being shown in search results, this method is also very clear to users. Please keep in mind that it generally does not make sense to set a geographic target if the same pages on your site target more than a single country (say, all German-speaking countries) — just write in that language and do not use the geotargeting setting (more on writing in other languages will follow soon!).
2. Server location (through the IP address of the server) is frequently near your users. However, some websites use distributed content delivery networks (CDNs) or are hosted in a country with better webserver infrastructure, so we try not to rely on the server location alone.
3. Other signals can give us hints. This could be from local addresses & phone numbers on the pages, use of local language and currency, links from other local sites, and/or the use of Google's Local Business Center (where available).

URL structures

Dealing with duplicate content on global websites

• "Thank you for this forum... Thank you to those that take the time to answer and care!"
• "I've only posted one question here, but have received a wealth of knowledge by reading tons of posts and answers. The time you experts put into helping people with their problems is very inspiring and my hat's off to each of you. Anyway, I just wanted to let you know that your services aren't going unnoticed and I truly appreciate the lessons."
• "Thank you very much cristina, what you told me has done the trick. I really appriciate the help as this has been bugging me for a while now and I didn't know what was wrong."
• "thank you ssssssssssoooo much kaleh. "
• "OK, Phil Payne big thanks to You! I have made changes and maybe people are starting to find me in G! Thanks to Ashley, I've started to make exclusive and relevant content for people."
• "If anything, it has helped me reflect on the sites and projects of days gone by so as to see what I could have done better - so that I can deliver that much more and better results going forward. I've learned that some things I had done right, were spot on, and other issues could have been handled differently, as well as a host of technical information that I've stored away for future use. Bottom Line: this forum rocks and is incredibly helpful."
• "I asked a handful of questions, got GREAT help while doing a whole lot of lurking, and now I've got a site that rocks!! (...) Huge thanks to all the Top Contributors, and a very special mention to WEBADO, who helped me a TON with my .htaccess file."
• "Over the years of reading (and sometimes contributing) to this forum I think it has helped to remove many false assumptions and doubts over Google's ranking systems. Contrary to what many have said I verily believe Google can benefit small businesses. Keep up the good work. "
• "The forum members are awesome and are a most impressive bunch. Their contribution is immeasurable as it is huge. Not only have they helped Google in their success as a profitable business entity, but also helped webmasters both aspiring and experienced. There is also an engender(ment) of "family" or "belonging" in the group that has transcended the best and worst of times (Current forum change still TBD :-) ). We can agree, disagree and agree to disagree but remain respectful and civil (Usually :-) )."
• "Hi Redleg, Thank you very much for all of the information. Without your help, I don't think I would ever have known how to find the problem. "
• "What an amazing board. Over the last few days I have asked 1 question and recieved a ton of advice mainly from Autocrat. "
• "A big thank you to the forum and the contributors that helped me get my site on Google . After some hassle with my web hosters and their naff submission service, issues over adding pages Google can see, issues over Sitemaps, I can now say that when I put my site name into the search and when i put in [custom made watch box], for instance, my site now comes up."
• "Thank you Autocrat! You are MAGNIFICENT! (...) I am your biggest fan today. : ) Imagine Joe Cocker singing With a Little Help from My Friends...that's my theme song today."
• "I've done a lot of reading since then and I've learned more in the last year than I learned in the previous 10. When I stumbled into this forum I had no idea what I was getting into but finding this forum was a gift from God! Words cannot express the amount of gratitude I feel for the help you have given me and I wish I could repay you some how.... I don't mean to sound so mushy, but I write this with tears in my eyes and I am truly, truly grateful..."

• Choose your preferred domain
  When confronted with duplicate content, search engines will generally take one version and filter the others out. This can also happen when multiple domain names are involved, so while search engines are generally pretty good at choosing something reasonable, many webmasters prefer to make that decision themselves.

• Reduce in-site duplication
  Before starting on cross-site duplicate content questions, make sure to handle duplication within your site first.

• Enable crawling and use 301 (permanent) redirects where possible
  Where possible, the most important step is often to use appropriate 301 redirects. These redirects send visitors and search engine crawlers to your preferred domain and make it very clear which URL should be indexed. This is generally the preferred method as it gives clear guidance to everyone who accesses the content. Keep in mind that in order for search engine crawlers to discover these redirects, none of the URLs in the redirect chain can be disallowed via a robots.txt file. Don't forget to handle your www / non-www preference with appropriate redirects and in Webmaster Tools.

• Use the cross-domain rel="canonical" link element
  There are situations where it's not easily possible to set up redirects. This could be the case when you need to move your website from a server that does not feature server-side redirects. In a situation like this, you can use the rel="canonical" link element across domains to specify the exact URL of whichever domain is preferred for indexing. While the rel="canonical" link element is seen as a hint and not an absolute directive, we do try to follow it where possible.

• My site's been hacked - now what?


• A quick security checklist for webmasters


• Message center warnings for hackable sites


• Best practices against hacking

• Minimal changes are required as the website grows


• Users and search engines see the same content (no cloaking)


• Search engines can send users directly to the AJAX URL (not to a static copy)


• Site owners have a way of verifying that their AJAX website is rendered correctly and thus that the crawler has access to all the content

• Slightly modify the URL fragments for stateful AJAX pages
  Stateful AJAX pages display the same content whenever accessed directly. These are pages that could be referred to in search results. Instead of a URL like http://example.com/page?query#state we would like to propose adding a token to make it possible to recognize these URLs: http://example.com/page?query#[FRAGMENTTOKEN]state . Based on a review of current URLs on the web, we propose using "!" (an exclamation point) as the token for this. The proposed URL that could be shown in search results would then be: http://example.com/page?query#!state.


• Use a headless browser that outputs an HTML snapshot on your web server
  The headless browser is used to access the AJAX page and generates HTML code based on the final state in the browser. Only specially tagged URLs are passed to the headless browser for processing. By doing this on the server side, the website owner is in control of the HTML code that is generated and can easily verify that all JavaScript is executed correctly. An example of such a browser is HtmlUnit, an open-sourced "GUI-less browser for Java programs.


• Allow search engine crawlers to access these URLs by escaping the state
  As URL fragments are never sent with requests to servers, it's necessary to slightly modify the URL used to access the page. At the same time, this tells the server to use the headless browser to generate HTML code instead of returning a page with JavaScript. Other, existing URLs - such as those used by the user - would be processed normally, bypassing the headless browser. We propose escaping the state information and adding it to the query parameters with a token. Using the previous example, one such URL would be http://example.com/page?query&[QUERYTOKEN]=state . Based on our analysis of current URLs on the web, we propose using "_escaped_fragment_" as the token. The proposed URL would then become http://example.com/page?query&_escaped_fragment_=state .


• Show the original URL to users in the search results
  To improve the user experience, it makes sense to refer users directly to the AJAX-based pages. This can be achieved by showing the original URL (such as http://example.com/page?query#!state from our example above) in the search results. Search engines can check that the indexable text returned to Googlebot is the same or a subset of the text that is returned to users.

1. Recognize duplicate content on your website.
   The first and most important step is to recognize duplicate content on your website. A simple way to do this is to take a unique text snippet from a page and to search for it, limiting the results to pages from your own website by using a site:query in Google. Multiple results for the same content show duplication you can investigate.
   


2. Determine your preferred URLs.
   Before fixing duplicate content issues, you'll have to determine your preferred URL structure. Which URL would you prefer to use for that piece of content?
   


3. Be consistent within your website.
   Once you've chosen your preferred URLs, make sure to use them in all possible locations within your website (including in your Sitemap file).
   


4. Apply 301 permanent redirects where necessary and possible.
   If you can, redirect duplicate URLs to your preferred URLs using a 301 response code. This helps users and search engines find your preferred URLs should they visit the duplicate URLs. If your site is available on several domain names, pick one and use the 301 redirect appropriately from the others, making sure to forward to the right specific page, not just the root of the domain. If you support both www and non-www host names, pick one, use the preferred domain setting in Webmaster Tools, and redirect appropriately.
   


5. Implement the rel="canonical" link element on your pages where you can.
   Where 301 redirects are not possible, the rel="canonical" link element can give us a better understanding of your site and of your preferred URLs. The use of this link element is also supported by major search engines such as Ask.com, Bing and Yahoo!.
   


6. Use the URL parameter handling tool in Google Webmaster Tools where possible.
   If some or all of your website's duplicate content comes from URLs with query parameters, this tool can help you to notify us of important and irrelevant parameters within your URLs. More information about this tool can be found in our announcement blog post.
   

• Google and the other search engines which are a part of Sitemaps.org now support up to 50,000 child Sitemaps for Sitemap index files (instead of the previous 1,000). This allows large sites to submit a theoretical maximum of 2.5 billion URLs with a single Sitemap Index URL (oh, and if you need more, you can always submit multiple Sitemap index files). 


• The Webmaster Tools design update now shows you all Sitemap files that were submitted for your verified website. This is particularly useful if you have multiple owners verified in Webmaster Tools or if you are submitting some Sitemap files via HTTP ping or through your robots.txt file.


• The indexed URL count in Webmaster Tools for your Sitemap files is now even more precise.


• For the XML developers out there, we've updated the XSD schemas to allow Sitemap extensions. The new schema helps webmasters to create better Sitemaps by verifying more features. By validating Sitemap files with the new schema, you can be more confident that the Sitemap files are correct.


• Do I need to mention that Sitemap file processing is much faster than ever before? We've drastically reduced the average time from submitting a Sitemap file to processing it and showing some initial data in Webmaster Tools. 

About Google Sitemap Generator

• XML Sitemaps for Web Search according to the sitemaps.org standard


• Mobile Sitemaps for mobile-friendly websites


• Code Search Sitemaps for source code that you make available to users

Getting started

Sitemap file formats supported by Google

• XML Sitemap files for web pages - Use these files to submit all of your web pages (this is the preferred format for web pages). While not all search engines may support the Sitemap types listed below, the XML Sitemap for web pages is supported by all search engines of sitemaps.org.
  


• RSS 2.0 and Atom 1.0 feeds for web pages - Many blogs create these automatically.


• Text files with web page URLs - If you can't automatically create one of the above formats, you can create a text file with your URLs in it.
  
  


• XML Sitemap files for Video Search - Videos on your website can be indexed and made available for Google Video Search.


• Media-RSS feeds for Video Search - mRSS feeds are used by various other systems, we can use these for Google Video Search as well.
  
  


• XML Sitemap files for Google Code Search - If you make programming samples or code available to your users, you can submit these for Google Code Search.


• XML Sitemap files for mobile web pages - Using this kind of format allows us to recognize content that has been optimized for mobile devices (please note that there was recently a small change in the format).


• XML Sitemap files for geo-data - If you have geographic data on your website in the form of KML or GeoRSS files, please let us know about these files.


• XML Sitemap files for News - News websites can submit their news content in this special Sitemap format (please note that you must first register with Google News before these files are processed).

Submitting your Sitemap files to Google

• Using Google Webmaster Tools
  Submitting your Sitemap files through Google Webmaster Tools is the preferred way of letting us know about them. The main advantage of doing it this way is that you'll always have direct feedback about how your Sitemap files were downloaded (were we able to reach your server?), how they were recognized (were they in the right format?) and what happened to the web pages listed in them (how many were indexed?). To submit your Sitemap files, make sure that your website is verified in Webmaster Tools, then go to "Sitemaps" in Webmaster Tools and enter the file name of your Sitemap(s).
  
  Sometimes it makes sense to keep your Sitemap file on a different server / domain name. To submit Sitemap files like that, you must verify ownership of both sites in Webmaster Tools and submit the Sitemap on the appropriate site. For instance, if your Sitemap file for http://www.example.com is kept on http://sitemap-files.example.com/ then you need to verify ownership of both sites and then submit the Sitemap file under http://sitemap-files.example.com (even though the URLs listed in it are for http://www.example.com). For more information, please see our Help Center topic on submitting Sitemap files for multiple sites.


• Listing Sitemap files in the robots.txt file
  Another way of submitting a Sitemap file is to specify the URL in your robots.txt file. If you use this method of submitting a Sitemap file, it will be found by all search engines that support the Sitemaps protocol (although not all of them support the extensions listed above). Since you can specify the full URL of your Sitemap file in the robots.txt file, this method also allows you to store your Sitemap file on a different domain. Keep in mind that while Sitemap files submitted this way are processed on our side, they will not be automatically listed in your Webmaster Tools account. In order to receive feedback on your files, we recommend adding them manually to your account as well.


• Using an HTTP "ping"
  If your Sitemap files are generated automatically, a convenient way to submit (and re-submit) them is to access the "ping" URL for Google Sitemaps. This URL includes the URL of your Sitemap file. For more information on the "ping" URL for your website, please see the Help Center article on Updating a Sitemap. Feel free to "ping" this URL whenever you update your Sitemap file - we'll know to pick it up and process it again. If you also have your Sitemap file registered in Webmaster Tools, we'll update the status there as well. This method is also valid if your Sitemap file is kept on a different server, but you must still verify both sites in Webmaster Tools as previously mentioned.
  
  Search engines that are a members of sitemaps.org support a similar way of submitting general web Sitemap files.

1. To include highly relevant content in Google's search index. This provides a better experience for Google users who may not have known that content existed.


2. To provide a promotion and discovery opportunity for publishers with restricted content.

• All users who click a Google search result to arrive at your site should be allowed to see the full text of the content they're trying to access.


• The page displayed to all users who visit from Google must be identical to the content that is shown to Googlebot.


• If a user clicks to a multi-page article, the user must be able to view the entire article. To allow this, you could display all of the content on a single page—you would need to do this for both Googlebot and for users. Alternately, you could use cookies to make sure that a user can visit each page of a multi-page article before being asked for registration or payment.

• webado (Christina from www.webado.net)
• Phil Payne (www.isham-research.co.uk)
• cass-hacks (Craig from cass-hacks.com)
• abracadabra (Tim from www.aitechsolutions.net)
• Aaron Pratt (www.ne-design.net)
• Cristina (www.asymptoticdesign.com)
• Robbo (Vince from www.tesol-direct.com)
• JLH (John Honeck from www.jlh-design.com)
• dockarl (Matt James from www.utheguru.com)
• IceGiant (Sasch from www.icegiant.co.uk)
• MrGamma (James from www.sitebyjames.com)
• The anonymous and helpful seo101

The latest version of Google Toolbar for Internet Explorer (beta) just added a neat feature to help users arrive at your website, or at least see your content, even when things go awry.

It's frustrating for your users to mistype your URL and receive a generic "404 - Not Found" or try to access a part of your site that might be down.

Regardless of your site being useful and information-rich, when these issues arise, most users just move on to something else.  The latest release of Google Toolbar, however, helps users by detecting site issues and providing alternatives.


Website Optimizer or Website Optimiser? The Toolbar can help you find it even if you try "google.cmo" instead of "google.com".

3 site issues detected by Google Toolbar

1. 404 errors with default error pages
   When a visitor tries to reach your content with an invalid URL and your server returns a short, default error message (less than 512 bytes), the Toolbar will suggest an alternate URL to the visitor. If this is a general problem in your website, you will see these URLs also listed in the crawl errors section of your Webmaster Tools account.
   
   If you choose to set up a custom error page, make sure it returns result code 404. The content of the 404 page can help your visitors to understand that they tried to reach a missing page and provides suggestions regarding how to find the content they were looking for. When a site displays a custom error page the Toolbar will no longer provide suggestions for that site. You can check the behavior of the Toolbar by visiting an invalid URL on your site with the Google Toolbar installed.
   
   
2. DNS errors
   When a URL contains a non-existent domain name (like www.google.cmo), the Toolbar will suggest an alternate, similar looking URL with a valid domain name. 
   
   
3. Connection failures
   When your server is unreachable, the Google Toolbar will automatically display a link to the cached version of your page. This feature is only available when Google is not explicitly forbidden from caching your pages through use of a robots meta tag or crawling is blocked on the page through the robots.txt file. If your server is regularly unreachable, you will probably want to fix that first; but it may also be a good idea to check the Google cache for your pages by looking at the search results for your site.
   

Suggestions provided by the Google Toolbar

• A link to the corrected URL
  When the Toolbar can find the most probable, active URL to match the user's input (or link they clicked on), it will display it right on top as a suggestion. The correction can be somewhere in the domain name, the path or the file name (the Toolbar does not look at any parameters in the URL).
  
  
• A link to the cached version of the URL
  When Toolbar recognizes the URL in the Google cache, it will display a link to the cached version. This is particularly useful when the user can't access your pages for some reason. As mentioned above, Google may cache your URLs provided you're not explicitly forbidding this through use of a robots meta tag or the robots.txt file.
  
  
• A link to the homepage or HTML site map of your site
  Sometimes going to the homepage or a site map page is the best way to find the page that a user is really looking for. Site map pages (these are not XML Sitemap files) are generally recognized based on the file name; if the Toolbar can find something called "sitemap.html" or similar, this page will probably be recognized as the site map page. Don't worry if your site map page is called something else; if a user decides to go to your homepage, they'll probably find it right away even if the Toolbar doesn't spot it.
  
  
• A link to a higher level folder
  Sometimes the homepage or site map page is too far out and the user would be better off just going one step up in the hierarchy. When the Toolbar can recognize that your site's structure is based on folders and sub-folders, it may suggest a page one step back.
  
  
• A search within your site for keywords found in the URL
  It's a good practice to use descriptive URLs. If the Toolbar can recognize keywords within the URL which the user tried to access, it will link to a site-search with those keywords. Even if the URL has changed significantly in the meantime, the search may be able to find similar content based on those keywords. For instance, if the URL was http://example.com/party-gifts/holidays/ it will suggest a search for the words "party", "gifts" and "holidays" within the site example.com.
  
  
• An open Google search box
  If all else fails, there's always a chance that similar content already exists elsewhere on the web. The Google web search can help your users to find it - the Toolbar will help you by adding the keywords found in the URL to the search box.

In writing and maintaining accurate meta tags (e.g., descriptive titles and robots information), you help Google to more accurately crawl, index and return your site in search results. Meta tags provide information to all sorts of clients, such as browsers and search engines. Just keep in mind that each client will likely only interpret the meta tags that it uses, and ignore the rest (although they might be useful for other reasons).

Here's how Google would interpret meta tags of this sample HTML page:

<meta name="description" content="A description of the page">
This tag provides a short description of the page. In some situations this description is used as a part of the snippet shown in the search results. For more information, please see our blog post "Improve snippets with a meta description makeover" and the Help Center article "How do I change my site's title and description?" While the use of a description meta tag is optional and will have no effect on your rankings, a good description can result in a better snippet, which in turn can help to improve the quality and quantity of visitors from our search results.

<title>The title of the page</title>
While technically not a meta tag, this tag is often used together with the "description." The contents of this tag are generally shown as the title in search results (and of course in the user's browser when visiting the page or viewing bookmarks). Some additional information can be found in our blog post "Target visitors or search engines?", especially under "Make good use of page titles."

<meta name="robots" content="…, …">
<meta name="googlebot" content="…, …">
These meta tags control how search engines crawl and index the page. The "robots" meta tag specifies rules that apply to all search engines, the "googlebot" meta tag specifies rules that apply only to Google. Google understands the following values (when specifying multiple values, separate them with a comma):

• noindex: prevents the page from being indexed (see "Block or remove pages using meta tags")
• nofollow: don't follow links from this page when looking for new pages to crawl (also see "Block or remove pages using meta tags")
• nosnippet: don't show a snippet of this page when displaying it in the search results (see "Prevent or remove snippets")
• noodp: don't use text from ODP (The Open Directory Project a.k.a. dmoz.org) to generate a title or snippet for this page (see "How do I change my site's title and description?")
• noarchive: don't display a "Cached" link for this page in the search results (see "Prevent or remove cached pages")
• unavailable_after:[date]: remove this page from the search results after the specified date and time (see "Robots Exclusion Protocol: now with even more flexibility")

The default rule is "index, follow" -- this is used if you omit this tag entirely or if you specify content="all." Additional information about the "robots" meta tag can be found in "Using the robots meta tag." As a side-note, you can now also specify this information in the header of your pages using the "X-Robots-Tag" HTTP header directive. This is particularly useful if you wish to fine-tune crawling and indexing of non-HTML files like PDFs, images or other kinds of documents.

<meta name="google" content="notranslate">
When we recognize that the contents of a page are not in the language that the user is likely to want to read, we often provide a link in the search results to an automatic translation of your page. In general, this gives you the chance to provide your unique and compelling content to a much larger group of users. However, there may be situations where this is not desired. By using this meta tag, you can signal that you do not wish for Google to provide a link to a translation for this page. This meta tag generally does not influence the ranking of the page for any particular language. More information can be found in the "Google Translate FAQ".

<meta name="verify-v1" content="…">
This Google webmaster tools-specific meta tag is used on the top-level page of your site to verify ownership of a site in webmaster tools (alternatively you may upload an HTML file to do this). The content value you put into this tag is provided to you in your webmaster tools account. Please note that while the contents of this meta tag (including upper and lower case) must match exactly what is provided to you, it does not matter if you change the tag from XHTML to HTML or if the format of the tag matches the format of your page. For details, see "How do I verify my site by adding a meta tag to my site's home page?"

<meta http-equiv="Content-Type" content="…; charset=…">
This meta tag defines the content-type and character set of the page. When using this meta tag, make sure that you surround the value of the content attribute with quotes; otherwise the charset attribute may be interpreted incorrectly. If you decide to use this meta tag, it goes without saying that you should make sure that your content is actually in the specified character set. "Google Webauthoring Statistics" has interesting numbers on the use of this meta tag.

<meta http-equiv="refresh" content="…;url=…">
This meta tag sends the user to a new URL after a certain amount of time, sometimes used as a simple form of redirection. This kind of redirect is not supported by all browsers and can be confusing to the user. If you need to change the URL of a page as it is shown in search engine results, we recommended that you use a server-side 301 redirect instead. Additionally, W3C's "Techniques and Failures for Web Content Accessibility Guidelines 2.0" lists it as being deprecated.

(X)HTML and Capitalization
Google can read both HTML and XHTML-style meta tags (regardless of the code used on the page). In addition, upper or lower case is generally not important in meta tags -- we treat <TITLE> and <title> equally. The "verify-v1" meta tag is an exception, it's case-sensitive.

revisit-after Sitemap lastmod and changefreq
Occasionally webmasters needlessly include "revisit-after" to encourage a search engine's crawl schedule, however this meta tag is largely ignored. If you want to give search engines information about changes in your pages, use and submit an XML sitemap. In this file you can specify the last-modified date and the change-frequency of the URLs on your site.

If you're interested in more examples or have questions about the meta tags mentioned above, jump into our Google Webmaster Help Group and join the discussion.