Google Data

Hello again everyone,

Today I wanted to go over one of the interesting things that you can do with OpenSocial using templating and data pipelining. Since these were introduced in version 0.9 of the specification, this functionality will not be available if:

• The container that you are using does not support OpenSocial version 0.9 or later.
• The container that you are using does not support data pipelining or templating.

With that said, let's get right down to an example. For this example, we're going to go through a few steps in order to scrape and display the recent headlines from Reddit:

• We will use data pipelining to load in the headlines from Reddit.
• The URL of the data pipe will be a dynamic call to the Yahoo! Query Language (YQL). YQL will be used to scrape the HTML from the page, then use an xpath to drill down to the repeating anchor tags that denote the headlines.
• Lastly, from the JSON data that is returned back from YQL into the data pipe, we're going to use OpenSocial templating to display out the headline results.

Let's take a look at the gadget code behind this:

xml version="1.0" encoding="utf-8"?>

Module>

   ModulePrefs title="Reddit Headline Fetch"

      title_url="http://www.jcleblanc.com"

      description="Obtains reddit.com headlines via YQL"

      author="Jonathan LeBlanc">

      Require feature="opensocial-0.9"/>

      Require feature="opensocial-data" />

      Require feature="opensocial-templates" />

   ModulePrefs>

   Content type="html">

   

   

   

   ]]>

   Content>

Module>

Let's break down the components of this gadget. First let's start with the meta data within the node. There are three features that we will need to require:

First we have the OpenSocial version that we want to include. For our needs we can include version 0.9 of the specification since that will allow us to include the other features we need, namely our second and third Require statements for opensocial-data (Data Pipelining) and opensocial-templates (Templating). These three will give us the features that we need for the example.

Next we have the innards of the node. Let's explore the first block:

This is our data pipelining section. When the gadget loads in the container that we are running it in, this section will issue an HTTP request to populate an object with the return value of the request. In our case this will be a GET request to the YQL service.

To define a data pipe, we set the type of the script block to text/os-data and then define the os XML namespace. Within the script block, we set the object that we want the data returned to to be "reddit". We then issue a data pipelining HTTP GET request to the YQL service, using the href attribute. This URL used includes the following components:

• The public YQL URL: http://query.yahooapis.com/v1/public/yql
• q: The YQL query to run, placed as a query string parameter.
• format: The format to return our data in, in our case JSON. This is also placed as a query string parameter.

If we look at the query that we are using, we can see what the components of the request are:

  SELECT * FROM html WHERE url="http://www.reddit.com"

  AND xpath='//a[@class="title"]'

YQL uses a query syntax much akin to SQL. In our request we are using the html table, which defines request functionality to allow us to scrape data from an HTML page and then apply an xpath to drill down to the nodes on the page that we want to capture. In our case, we are capturing all data from http://www.reddit.com where the xpath (or nodes that we want to return) is any anchor tag that contains a class of title. If we explore the source code of http://www.reddit.com we can see that each title on the page is indeed an anchor tag with a class of title. At the end of the request the reddit object should now contain the return value from the request or, in other words, the headlines from the Reddit homepage.

Now let's look at the templating script block:

This is the block that will define the template for displaying our results. To define a templating script block, we define the type to be "text/os-template" and then we can include the object (in our case from the data pipelining request) that we want to make available to the template - we set this to "reddit" - the same value that we set as the key of the data pipelining request.

Now we can simply define the HTML that we want to use, with the variables, to display our results. For this simple use case we'll simply create an unordered list with all of the headlines displayed as anchor tag links. In OpenSocial templating, we can insert the value from a data source by including the variable that we want to act upon, displaying it in the format ${VARIABLE}. For instance, in our data pipe if we defined our key to be "foo" and in the foo object we have a sub-element named "bar", we could display it using ${foo.bar}. Another aspect about templating that we're going to be taking advantage of is repeaters. This will allow us to repeat a block of code for each object available, and is done so by adding a repeat="${OBJECT}" attribute on the block that should be repeated, in this case the

tag. This will create a list item for each headline we encounter.

Now, if we look at the HTML block knowing this, we can see that we are creating a root unordered list node, then creating a list item for each headline we encounter in the Reddit headlines object. The headlines are embedded in the object at reddit.content.query.results.a. Within each list item we display an anchor tag. The href of the anchor is set to that of the headline link and the text of the link is set to the headline that we get back from the query.

Once the gadget has rendered you should be presented with a very unstyled unordered list, which you may then apply CSS on to. Using this type of setup with a service like YQL, data pipelining, and templating, you can create gadgets that render very dynamic content while reducing the amount of coding that is needed.

Jonathan LeBlanc (@jcleblanc)
Jonathan LeBlanc is a principal developer evangelist with X.commerce. Jonathan has been a member of the OpenSocial community for over three years and is the author of O'Reilly's "Programming Social Applications".