• Google wants to serve up unique results and does a great job of picking a version of your content to show if your sites includes duplication. If you don't want to worry about sorting through duplication on your site, you can let us worry about it instead.
• Duplicate content doesn't cause your site to be penalized. If duplicate pages are detected, one version will be returned in the search results to ensure variety for searchers.
• Duplicate content doesn't cause your site to be placed in the supplemental index. Duplication may indirectly influence this however, if links to your pages are split among the various versions, causing lower per-page PageRank.

• Adding a rel="nofollow" attribute to the href tag
• Redirecting the links to an intermediate page that is blocked from search engines with a robots.txt file

Quality guidelines - specific guidelines

• Avoid hidden text or hidden links.
• Don't use cloaking or sneaky redirects.
• Don't send automated queries to Google.
• Don't load pages with irrelevant keywords.
• Don't create multiple pages, subdomains, or domains with substantially duplicate content.
• Don't create pages that install viruses, trojans, or other badware.
• Avoid "doorway" pages created just for search engines, or other "cookie cutter" approaches such as affiliate programs with little or no original content.
• If your site participates in an affiliate program, make sure that your site adds value. Provide unique and relevant content that gives users a reason to visit your site first.

• We've expanded the number of phrases we show to 200.
• You can now see the variations of each phrase (for instance, with different capitalization and punctuation).
• More sites now have access to the anchor phrase report. So, if you didn't have this report before, you may have it now.
• We've brought back the report showing the most common individual words in anchor text (you asked; we delivered!).
• We've expanded the number of common words in anchor text and common words in your site that we show to 100 each.

• The URL must return a status code of either 404 or 410.
• The URL must be blocked by the site's robots.txt file.
• The URL must be blocked by a robots meta tag.

• If you block a page with robots.txt, Googlebot will never crawl the page and will never read any meta tags on the page.
• If you allow a page with robots.txt but block it from being indexed using a meta tag, Googlebot will access the page, read the meta tag, and subsequently not index it.

• NOINDEX - prevents the page from being included in the index.
  
• NOFOLLOW - prevents Googlebot from following any links on the page. (Note that this is different from the link-level NOFOLLOW attribute, which prevents Googlebot from following an individual link.)
  
• NOARCHIVE - prevents a cached copy of this page from being available in the search results.
  
• NOSNIPPET - prevents a description from appearing below the page in the search results, as well as prevents caching of the page.
  
• NOODP - blocks the Open Directory Project description of the page from being used in the description that appears below the page in the search results.
• NONE - equivalent to "NOINDEX, NOFOLLOW".
  

• Check the Crawl errors section of webmaster tools for any pages Googlebot couldn't access due to server or other errors. If Googlebot can't access the pages, they won't be indexed and visitors likely can't access them either.
• Make sure your robots.txt file doesn't accidentally block search engines from content you want indexed. You can see a list of the files Googlebot was blocked from crawling in webmaster tools. You can also use our robots.txt analysis tool to make sure you're blocking and allowing the files you intend.
• Check the Googlebot activity reports to see how long it takes to download a page of your site to make sure you don't have any network slowness issues.
• If pages of your site require a login and you want the content from those pages indexed, ensure you include a substantial amount of indexable content on pages that aren't behind the login. For instance, you can put several content-rich paragraphs of an article outside the login area, with a login link that leads to the rest of the article.
• How accessible is your site? How does it look in mobile browsers and screen readers? It's well worth testing your site under these conditions and ensuring that visitors can access the content of the site using any of these mechanisms.

Make sure your content is viewable

• Check out your site in a text-only browser or view it in a browser with images and Javascript turned off. Can you still see all of the text and navigation?
• Ensure the important text and navigation in your site is in HTML, not in images, and make sure all images have ALT text that describe them.
• If you use Flash, use it only when needed. Particularly, don't put all of the text from your site in Flash. An ideal Flash-based site has pages with HTML text and Flash accents. If you use Flash for your home page, make sure that the navigation into the site is in HTML.

Be descriptive

• Make sure each page has a unique title tag and meta description tag that aptly describe the page.
• Make sure the important elements of your pages (for instance, your company name and the main topic of the page) are in HTML text.
• Make sure the words that searchers will use to look for you are on the page.

Keep the site crawlable

• If possible, avoid frames. Frame-based sites don't allow for unique URLs for each page, which makes indexing each page separately problematic.
• Ensure the server returns a 404 status code for pages that aren't found. Some servers are configured to return a 200 status code, particularly with custom error messages and this can result in search engines spending time crawling and indexing non-existent pages rather than the valid pages of the site.
• Avoid infinite crawls. For instance, if your site has an infinite calendar, add a nofollow attribute to links to dynamically-created future calendar pages. Each search engine may interpret the nofollow attribute differently, so check with the help documentation for each. Alternatively, you could use the nofollow meta tag to ensure that search engine spiders don't crawl any outgoing links on a page, or use robots.txt to prevent search engines from crawling URLs that can lead to infinite loops.
• If your site uses session IDs or cookies, ensure those are not required for crawling.
• If your site is dynamic, avoid using excessive parameters and use friendly URLs when you can. Some content management systems enable you to rewrite URLs to friendly versions.

• robots.txt analysis tool
• page with the highest PageRank by month
• common words in your site's content and in anchor text to your site

• showing the top position of your site for your top queries
• top mobile queries
• download options for Sitemaps data, stats, and errors
  

• meta tag verification
• notification of violations to the webmaster guidelines
  
• reinclusion request form and spam reporting form
  
• indexing information (can we crawl your home page? is your site indexed?)

• increased the number of crawl errors shown to 100% within the last two weeks
• Increased the number of Sitemaps you can submit from 200 to 500
• Expanded query stats so you can see them per property and per country and made them available for subdirectories
  
• Increased the number of common words in your site and in links to your site from 20 to 75
• Added Adsbot-Google to the robots.txt analysis tool
  

• Preferred domain control
  
• Site verification management
  
• Downloads of query stats for all subfolders

• Crawl rate control
  
• Googlebot activity reports
• Opting in to enhanced image search
  
• Display of the number of URLs submitted via a Sitemap
  

• Don't put the bulk of your text in images. It may sound simple, but the best thing you can do is to put your text into well, text. Reserve images for graphical elements. If all of the text on your page is in an image, it becomes inaccessible.
• Take advantage of alt tags for all of your images. Make sure the alt text is descriptive and unique. For instance, alt text such as "picture1" or "logo" doesn't provide much information about the image. "Charting the path of stock x" and "Company Y" give more details.
• Don't overload your alt text. Be descriptive, but don't stuff it with extra keywords.
• It's important to use alt text for any image on your pages, but if your company name, navigation, or other major elements of your pages are in images, alt text becomes especially important. Consider moving vital details to text to ensure all visitors can view them.
• Look at the image-to-text ratio on your page. How much text do you have? One way of looking at this is to look at your site with images turned off in your browser. What content can you see? Is the intent of your site obvious? Do the pages convey your message effectively?

• pages it already knows about
  
• links from other web pages (within your site and on other sites)
• pages listed in your Sitemap file

• http://www.example.com/
• http://www.example.com/index.html
• http://example.com
• http://example.com/index.html

• http://www.example.com/mypage.html#heading1
• http://www.example.com/mypage.html#heading2
• http://www.example.com/mypage.html#heading3

• http://www.example.com/furniture?type=chair&brand=123
• http://www.example.com/hotbuys?type=chair&brand=123

Consider this text:

“We have hundreds of workshops and classes available. You can choose the workshop that is right for you. Spend an hour or a week in our relaxing facility.”

Will this site show up for searches for [cooking classes] or [wine tasting workshops] or even [classes in Seattle]? It may not be as obvious to visitors (and search engine bots) what your page is about as you think.

Along those same lines, does your content use words that people are searching for? Does your site text say “check out our homes for sale” when people are searching for [real estate in Boston]?

Googlebot activity reports
Check out these cool charts! We show you the number of pages Googlebot's crawled from your site per day, the number of kilobytes of data Googlebot's downloaded per day, and the average time it took Googlebot to download pages. Webmaster tools show each of these for the last 90 days. Stay tuned for more information about this data and how you can use it to pinpoint issues with your site.

Crawl rate control
Googlebot uses sophisticated algorithms that determine how much to crawl each site. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth.

We've been conducting a limited test of a new feature that enables you to provide us information about how we crawl your site. Today, we're making this tool available to everyone. You can access this tool from the Diagnostic tab. If you'd like Googlebot to slow down the crawl of your site, simply choose the Slower option.

• Tips from BlogHer | More tips
• Opting-out of Open Directory Project descriptions and titles in our search results
• Tips for non-US sites
• Information on robots.txt files | More tips on using a robots.txt file
• Improving your site's indexing and ranking 101
• Changing domain names | More about changing domain names

• Can multiple account holders view information for a site?
• About meta tag verification
• About the common words feature
• Changing your Sitemap when your site changes
• Do you have to include every page from your site in your Sitemap?
• If you try to verify and get the error: We've detected that your 404 (file not found) error page returns a status of 200 (OK) in the header
• Submitting mobile Sitemaps

• download individual tables of data by clicking the Download this table link.
• download stats for all subfolders on your site (for all search types and locations) by clicking the Download all query stats for this site (including subfolders) link.
• download all stats (including query stats) for all verified sites in your account by choosing Tools from the My Sites page, then choosing Download data for all sites and then Download statistics for all sites.

• When you set the preferred domain, we no longer will add the non-preferred version to your account.
• If you had previously added the non-preferred version to your account, you'll still see it listed there, but you won't be able to add a Sitemap for the non-preferred version.
• If you have already set the preferred domain and we had added the non-preferred version to your account, we'll be removing that non-preferred version from your account over the next few days.

• Consider all links that point to the site (whether those links use the www version or the non-www version) to be pointing at the version you prefer. This helps us more accurately determine PageRank for your pages.
• Once we know that both versions of a URL point to the same page, we try to select the preferred version for future crawls.
• Index pages of your site using the version you prefer. If some pages of your site are indexed using the www version and other pages are indexed using the non-www version, then over time, you should see a shift to the preference you've set.
  

1. Googlebot crawls a page on April 12, 2006.
2. Our cached version of that page notes that "This is G o o g l e's cache of http://www.example.com/ as retrieved on April 12, 2006 20:02:06 GMT."
3. Periodically, Googlebot checks to see if that page has changed, and each time, receives a Not-Modified response. For instance, on August 27, 2006, Googlebot checks the page, receives a Not-Modified response, and therefore, doesn't download the contents of the page.
4. On August 28, 2006, our cached version of the page still shows the April 12, 2006 date -- the date we last downloaded the page's contents, even though Googlebot last visited the day before.

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow: /cgi-bin/

User-agent: *
Disallow: /