Google Data

Posted by Josh Cohen, Senior Business Product Manager

As newspapers consider charging for access to their online content, some publishers have asked: Should we put up pay walls or keep our articles in Google News and Google Search? In fact, they can do both - the two aren't mutually exclusive. There are a few ways we work with publishers to make their subscription content discoverable. Today we're updating one of them, so we thought it would be a good time to remind publishers about some of their options.

Google has strict policies against what's known as cloaking: showing one web page to the crawler that indexes it but then a different page to a user. We do this so that users aren't deceived into clicking through to a site that's not what they were expecting. While the anti-cloaking policies are important for users, they do create some challenges for publishers who charge for content. Our crawlers can't fill out a registration or payment form to see what's behind a site's paywall, but they need access to the information in order to index it.

One way we overcome this is through a program called First Click Free. Participating publishers allow the crawler to index their subscription content, then allow users who find one of those articles through Google News or Google Search to see the full page without requiring them to register or subscribe. The user's first click to the content is free, but when a user clicks on additional links on the site, the publisher can show a payment or registration request. First Click Free is a great way for publishers to promote their content and for users to check out a news source before deciding whether to pay. Previously, each click from a user would be treated as free. Now, we've updated the program so that publishers can limit users to no more than five pages per day without registering or subscribing. If you're a Google user, this means that you may start to see a registration page after you've clicked through to more than five articles on the website of a publisher using First Click Free in a day. We think this approach still protects the typical user from cloaking, while allowing publishers to focus on potential subscribers who are accessing a lot of their content on a regular basis.

In addition to First Click Free, we offer another solution: We will crawl, index and treat as "free" any preview pages - generally the headline and first few paragraphs of a story - that they make available to us. This means that our crawlers see the exact same content that will be shown for free to a user. Because the preview page is identical for both users and the crawlers, it's not cloaking. We will then label such stories as "subscription" in Google News. The ranking of these articles will be subject to the same criteria as all sites in Google, whether paid or free. Paid content may not do as well as free options, but that is not a decision we make based on whether or not it's free. It's simply based on the popularity of the content with users and other sites that link to it.

These are two of the ways we allow publishers to make their subscription content discoverable, and we're going to keep talking with publishers to refine these methods. After all, whether you're offering your content for free or selling it, it's crucial that people find it. Google can help with that.