Google Data » Matt Wytock

Custom Search at the core of Google Site Search

Matt Wytock — Tue, 03 Jun 2008 08:47:00 +0000

Posted by: Matt Wytock and Vrishali Wagle, Software Engineers

Today, we announced Google Site Search, a hosted website search product that takes advantage of the Custom Search platform to offer high quality search to any website. Google Site Search integrates features that offer site visitors a search experience closer to what they're used to on Google.com. Here are a few of the new features:

Expanded coverage: We already search through all of your pages that are in the Google.com index. Tell us about more pages on your site by submitting a Sitemap. We'll crawl your Sitemap and add these pages to your custom search engine so that your search engine has the maximum coverage. (These additional pages will only be available to your custom search engine; your PageRank and Google.com rankings won't change in any way.)

Custom Synonyms: You can now define custom synonyms for your custom search engine. For example, you can define "cd" as meaning"certificate of deposit." When a visitor searches for "cd" on your search engine, we will return pages that contain either "cd" or "certificate of deposit." You can specify these synonyms in the XML definition of your search engine.

Date Biasing: Fine-tune the relevance of search results by specifying a bias for newer documents. We allow various levels of biasing, the highest of which approximately sorts by date.

Top Results Biasing: If you want the first N results to always match a refinement, you can specify that as a property of the refinement. We will try to fill up the top N positions with results matching that refinement before showing other relevant results.

We're not done with Custom Search and are always thinking about ways to add new features. Stay tuned, as we've only just begun.

Searching for Creative Commons licensed content

Matt Wytock — Thu, 24 Apr 2008 00:52:00 +0000

Posted by: Tania Bedrax-Weiss, Software Engineer

If you've ever wanted to let your users only see content that is free to use and share (such as through Creative Commons), here are a few tips on how to do it. Assuming you've indicated the CC license in your pages, you can now specify that you want only content with specific licensing terms to show up in your CSE. We've made available four labels for CSE creators to use based on the CC licenses:

free_use_share (by-nc-nd)
free_use_share_commercially (by-nd)
free_use_share_modify (by-nc-sa)
free_use_share_modify_commercially (by-sa)

You can either use one of these labels as a background label to restrict all of your content to these terms, or you can reference these labels in the facet items section of the CSE specification and they will be exposed to users as additional refinements in your CSE. Note that these are filter labels; you cannot boost according to these labels.

As you generate content, please keep in mind that there are a number of ways Creative Commons licenses may be added to an HTML doc:

with ... in the HTML head or body
using
using
using ... in a comment

If your documents use any of these methods, we likely already have licensing information. See for example: http://joshua.smcvt.edu/linearalgebra/ that illustrates method number 3:

...

and also illustrates method number 4:

See the file source for the complete text and don't forget to visit our Webmaster Central for more information.

Labels in Custom Search

Matt Wytock — Thu, 27 Mar 2008 14:49:00 +0000

Posted by: Phil Liao, Software Engineer

Last month we demonstrated how to specify URL patterns in your CSE. We now explain another powerful concept in Custom Search: labels, which are also known as refinements. Labels are typically displayed above the search results.

Let's revisit our I Love Veggies search engine. When the user searches for "eggplant", we display some categorical refinements above her search results such as: recipes, nutrition, farmer's markets, and growing tips. When she clicks the "recipes" refinement, for example, our CSE emphasizes eggplant recipes in her search results.

To make refinements, do the following:

1. Create refinement labels.

To add a refinement to your custom search engine, go to the Refinements tab in the control panel, then click the Add Refinement button. Enter a refinement label and its associated keywords, and choose how to prioritize the labeled sites. You can enter one of our predefined refinements or create one of your own.

2. Tag sites with refinement labels.

Once you have configured your refinements, go to the Sites tab in the control panel. Choose some sites (URL patterns) using the checkboxes, then choose a label from the Label actions drop-down list. You can attach any number of labels to each site. For example, if a site includes nutritional information with its recipes, you could label it with both the "recipes" and the "nutrition" refinements. Labeling your sites properly enables your users to find sites that are relevant to each refinement. In addition, by showing meaningful labels under each search result, you enable your users to find similar results quickly. The Custom Search Help Center has helpful hints on how to choose good refinements.

And this is how to create refinements and assign their labels to your sites using the Custom Search control panel. We'll show you advanced refinement management using XML in another post soon.

Making the most with AdSense

Matt Wytock — Tue, 18 Mar 2008 16:39:00 +0000

Posted by: Talia Brodecki, AdSense Product Marketing

If you're currently making money from CSE with AdSense, here are a few optimization tips to help you improve your current performance. (If you're not already earning money this way, scroll down to learn more about how you can start doing so, using your search box.)

1) Search box placement: Search boxes should be easy to find, typically under the header or on the left navigation bar, to encourage users to simply search for what they're looking for instead of spending time browsing through your content. Try to keep your search box in the same place on every page so users know where to return for future searches.

2) Two search boxes per page: For pages with a lot of content, we recommend placing two search boxes per page - one at the top of the page where users can immediately go to to find what they're looking for and one at the bottom of the page where users can quickly search for something else after reading the content on the page. You can track the performance of each by creating a custom channel.

3) iFRAME: By choosing to host the search results, you can integrate search results and ads within your site so users don't leave your site when they perform a search. If users don't find what they're looking for in the search results, they can easily click to another section of your site through your site's template.

4) Search box on search results page: Remember to place another search box on your search results page in the case that a user wants to perform another search.

5) Ad placement: For better monetization, we recommend choosing to place ads at the top & right of the search results for added visibility.

6) Appearance customization: Customize the search results with your site's colors to match the look and feel.

If you aren't using AdSense, we encourage you to sign up and get paid for every valid click on ads you're already showing. Just go to the Make Money link in the CSE control panel and fill out the short form to sign up for an account. If you already have an AdSense account, you can easily link the two to start earning money immediately.

Now then -- better go figure out what to buy with all that extra cash. ;)

Custom Search on Google Help Centers

Matt Wytock — Tue, 04 Mar 2008 23:09:00 +0000

Posted by: Christine Moschella, Online Operations

Custom Search Engines (CSEs) are powerful tools for searching over the pages of your site, as well as providing links to your favorite resources across the web. Because they provide great search functionality with the option to specify a subset of Google's web index, they're perfect for us to use in our Google Help Centers.

We've rolled out CSEs to many of our product Help Centers already, including the Custom Search Help Center, and we're also using it on our main Google Help page.

The Help Center search engines use several key features of CSEs such as refinements and labels to identify result sources, and Subscribed Links for certain help queries, like this one in the Analytics Help Center.

Because Help Center searches will now include content from product pages, developer documentation, Google blogs, and help groups, we hope that search will be an even more effective tool to help you find answers to your Google product questions. For example, failed searches - searches that don't return any results - decreased by 23% on the Analytics Help Center after we added a CSE.

Specifying patterns for your Custom Search Engine

Matt Wytock — Fri, 22 Feb 2008 00:24:00 +0000

Posted by: Vrishali Wagle, Software Engineer

Creating a basic Custom Search Engine (CSE) is very easy. You enter a list of sites, select a few basic preferences, and you are done, right? But in fact there's more to Custom Search -- consider it a very powerful way of building your own search engine on top of Google search. You can exclude sites, add labels for drill-down and even change the ranking of results for your search engine. In this blog post, we look at the basic element of Custom Search - URL patterns

URL patterns specify the part of the web you want to search or exclude from your search. Custom Search is based on approximation algorithms that use these patterns to give you your customized results.

Consider the "I Love Veggies" search engine that we created. Here's how the "I Love Veggies" search engine made use of patterns effectively:

Be very specific. Use the longest possible pattern for specifying a site. For example, in the "I Love Veggies" search engine, we wanted to search all of www.goveg.com, so we added "www.goveg.com/*" as a pattern. But we wanted to search only the vegetarian part of the "allrecipes.com" site. So instead of adding all of "allrecipes.com/*" we added the more specific "allrecipes.com/Recipes/Everyday-Cooking/Vegetarian/*".
Specify multiple pages in a site with a "*" at the end of the pattern. If you specify just "www.goveg.com", Custom Search will search just the single page http://www.goveg.com. You need to remember this only if you are write your XML file of annotations directly. If you are using the Control Panel, it automatically adds the "/*" at the end for you, unless you indicate otherwise.
Sometimes, you might have a few hosts on a domain with the same path that you want to search. In our example, we wanted to search "mideastfood.about.com/od/vegetarianrecipes/*" and "indianfood.about.com/od/vegetarianrecipes/*". In such a case it is better to specify these patterns individually instead of a very general "*.about.com/od/vegetarianrecipes/*" as more specific the patterns, better the approximation.
You can only use the * in the hostname at the beginning of the pattern and it can only represent a full token. For example, "*.about.com/*" is a valid pattern and so is "*.food.about.com/*". However, "*ood.about.com/*" is not valid, nor is "food.*.about.com/*".

Keep reading this blog for more tips and tricks as we develop our "I Love Veggies" search engine. If you have specific questions or feature requests you can visit our Help Center or ask a question on the Discussion group.

Displaying the most relevant refinements

Matt Wytock — Mon, 07 Jan 2008 19:04:00 +0000

Posted by: Vrishali Wagle, Software Engineer

Over the past year, a number of you have asked to show only the matching refinements above Custom Search results. Today we're announcing that very feature: You can have as many FacetItems as you like in the XML definition of your CSE and control the number that display on a search page using this feature. Just add a "top_refinements" attribute to your context definition's CustomSearchEngine tag.

Where n is the number of refinements you want to show.

For example, consider the context definition of a CSE I created: http://vrishali.googlepages.com/food_context.xml. It has sixteen FacetItems -- however, the top_refinements is set to 4. This means that it will display the top four matching refinements for every query. Check out the query chicken. Observe how the refinements displayed at the top are different for the query food safety.

Now consider the default implementation of the same CSE, that is without the top_refinements option: http://vrishali.googlepages.com/food_context_no_top_refinements.xml. All queries in this search engine will display the same refinements, for example chicken and food safety.

This feature is enabled for Linked CSEs as well as Google stored CSEs. To use it in your Google Stored CSE, download the context file in the "Advanced" tab of your control panel, add the "top_refinements" attribute to the CustomSearchEngine tag in your XML and upload the modified file.

If you don't want to display refinements at the top of the search results, turn them off by setting top_refinements="0". Your search results will still be labeled with the refinements that match.

As always, let us know what you like, or don't like, about this feature!

Ads now in harmony with search results

Matt Wytock — Mon, 17 Dec 2007 23:35:00 +0000

Posted by: Tom Duerig, Software Engineer

Many folks have pointed out that all the results on the page -- both the search results and the advertisements -- should be appropriately customized so that the ads are in sync with the search results. We are happy to tell you that this is now indeed the case. Keywords that you use to tune search results will also be used to tune the ads.

To demonstrate the power of keywords for tuning ads, we've created 3 search engines. Note the differences in ads when we search for [leash] on all three:

A search engine about children, with the keywords [child children]:

A search engine about dogs, with the keywords [dog dogs]:

And, a search engine about surfing, with the keywords [surf surfing]:

So, keywords are even more important than before to your search engine. Make sure that you choose the keywords that provide the best overall search experience!