Use your expertise – build a topical search engine

December 10th, 2013 | Published in Google Custom Search

Did you know that a topical search engine can help your users find content from more than a single domain? You can use your expertise to provide a delightful user experience targeting a particular topic on the Web.

There are two main types of engines built with Google Custom Search: site search and topical search. While site search is relatively straightforward - it lets you implement a search for a blog or a company website - topical search is an entirely different story.

Topical search engines focus on a particular topic and therefore usually cover a part of the Web that is larger than a single domain. Because of this topical engines need to be carefully fine-tuned to bring the best results to the users.

Kritikos

A nice example of a topical search engine is Kritikos - a search engine for visual media relevant to higher education developed by the Engineering Department at Liverpool University. Developed in its pilot phase exclusively for Engineering students, Kritikos is now able to add new academic subjects simply by changing one line of code in the API call.

Kritikos is using a mix of Custom Search tuning techniques to find the most relevant results and then matches the results with the data stored in the local node of the Learning Registry. The additional data is displayed as an overlay in the search results page, providing valuable reference for the engine’s users. This method allows Kritikos to also include a social mechanism into their results - e.g. it is able to show users votes or comments about the usefulness of particular resources. Here is what Andrew Green, the Technical Director of Kritikos from the University of Liverpool says about the project.

In developing Kritikos, we wanted to build upon existing search engines, rather than try and build our own. We discovered that, once we found the documentation, the Google Custom Search Engine API was incredibly easy to use. The call to the API, based on the user’s subject of interest, search terms and preferred media type, is constructed in only a dozen lines of code. The resulting JSON data was easily parsed in order to generate the thumbnail images that give Kritikos its visual impact.

This shows the first 3 results for the term query term “fracture”, as seen by an Engineering student. The icons in the top right corners indicate that other students have interacted with these resources in some way.

Here are the results for the same search term (“fracture”), this time as viewed by a Medical student.

Andrew Green

Technical Director, Kritikos

The University of Liverpool

You can read more about their adventures with Custom Search on the project’s blog.

From talking to developers working with Custom Search and looking at different engines out there on the Web, I learned a few trends present in many of the solutions. Here they are - a few things to keep in mind when building a topical search engine.

Choose your index wisely

There is a lot of mess on the Internet. You don’t want to distract your users and make them spend time on random sites. The point of making a topical search engine is to make it focused and targeted.

An index is a list of sites - or url patterns - that your engine is operating on. Choosing what should go into the engine’s index and what should be left out is a matter of expected quality. Sometimes even sites that are on topic can be left out of the index or de-prioritized in order to promote other content, that you - the author of the engine - consider more valuable to your users.

Guess your user intention

The usability rule - Don’t make me think - plays nicely with topical search engines. You build the engine to make your users life easier, finding the information faster and more efficient. Usually your engine is functioning in some particular context - it is included in a topical website, it is addressed to some specific audience etc. You can apply this context to the search process, to make your users more productive and happier with the experience. There are a number of techniques that allow you to add context to your users’ queries - for details check out the Topical CSE article on Google Developers site.

Provide delightful experience with custom rendering

Search does not need to be boring. You can add all sorts of fireworks to your search results - custom design, non-linear browsing experience, user interactions or displaying additional data.

Google Custom Search allows you to change almost any aspect of the presentation layer of the results - as long as it is not misleading to the users.

What is coming

Structured data is gaining more and more momentum on the Web. It is a way of expressing real-world concepts in a machine-readable format - for example information about places, events, businesses etc. It already powers some of Google’s products and features, like Rich snippets, Knowledge Graph panels or Google Now cards.

You can harness the power of structured data in a topical search engine by filtering results by a schema.org types and properties or creating your own custom rich snippets.

Stay tuned, we will post more about the integration of Google Custom Search and schema.org standard soon.

Learn how

In the Advanced Topics section of developer’s documentation, you can find an article about creating a Topical CSE. It describes in more detail the techniques needed to obtain the results discussed in this blog post.

If you have an interesting topical search engine and would like to share it, give us a shout at twitter at @googlecse, we’d love to see what you came up with. Happy coding!

Ewa Gasperowicz, Developer Programs Engineer

Google Data