Google Data » Jeff Bailey

Google Update Goes Open Source

Jeff Bailey — Fri, 10 Apr 2009 19:29:00 +0000

Keeping software up to date is very important. Not only does it mean that users will always have all the cool new features that we work so hard to develop; it also means that any bugs or security vulnerabilities can get fixed very quickly, everywhere that the software is installed. We're happy to let you know that we're sharing our updating software, Google Update, with everyone. Google Update is the shared infrastructure used by Google Chrome, Google Earth and other Google software on Microsoft Windows, to keep our products up to date on users computers.

We're releasing Google Update under its codename Omaha. Omaha's functionality allows us to automatically update software without interrupting or distracting the user, which makes for a better user experience. Omaha checks for updates in the background, when it won't interfere with the user, even if an application isn't running. Doing so means that we avoid using a computer's resources when it first starts, avoiding a common bottleneck in computer performance experience. Omaha does not perform updates when an application launches, because we understand people want to use the software when starting it up, not perform maintenance tasks first.

Use of Omaha allows us to add features seamlessly and address any bugs or security problems, all without concern that these updates will disrupt our users. Omaha allowed us to ship 12 versions of Chrome beta in 4 months, without requiring Chrome users to work hard to keep their browsers up to date. Such behavior is very useful for new features, but essential for security vulnerabilities. When software, particularly network-enabled software, has known vulnerabilities, it can become a platform for malware and/or spam distribution as described in this research paper. Keeping your software up to date can help other people too!

We're releasing the source code for Omaha in addition to recent enhancements to Omaha functionality, to provide both transparency and control around the update process. Since Google Update is always running on your system, there's no simple way to stop it, and since it's a fundamental part of the Google software that needs it, it's not explicitly installed. Some users can be surprised to find this program running, and at Google, we don't like disappointing our users. We've been working hard to address these concerns, and releasing the source code for Omaha is our attempt to make the purpose of Google Update totally transparent. Obviously, we understand that not everyone is both willing and able to read through our code, but we hope that those of you who do will confirm for the rest that Google Update's functionality serves well to keep your software up to date.

Finally, we also know that keeping software up to date is hard. So if you're thinking of developing your own auto-updater, or have already started, we hope that the code we are releasing today will be helpful to you! So far, Omaha supports many Google products for Windows, but there is no reason for it to only support Google products. We hope you'll find the source code and Developer Startup Guide useful, and we look forward to your feedback and participation in our Discussion Group.

Post by Myles Jordan, Software Engineering Team and Michael Smith, Product Management Team

OpenStreetMap’s New API Database Server

Jeff Bailey — Fri, 20 Feb 2009 03:19:00 +0000

By Ellen Ko, Open Source Team

A few weeks ago Mano Marks, a Geo Developer Advocate here at Google, informed the Open Source Programs Office of OpenStreetMap's server upgrade fundraiser. Recognizing the good work that OpenStreetMap does by providing free geographic data of the world, we were happy to respond. We donated 5,000 GBP to the cause, and along with contributions from around the world, OpenStreetMap exceeded their goal in a single week! Congratulations to OpenStreetMap for building such an active and devoted community.

If you are interested in a collaborative project to create a free editable map of the world, we encourage you to check out their site and see what all the excitement is about.

Emoji for Unicode: Open Source Data for the Encoding Proposal

Jeff Bailey — Wed, 26 Nov 2008 23:19:00 +0000

By Markus Scherer, Google Internationalization Engineering

Emoji (絵文字), or "picture characters", the graphical versions of :-) and its friends, are widely used and especially popular among Japanese cell phone users. Just last month, they became available in Gmail ― see the team's announcement: A picture is worth a thousand words.

These symbols are encoded as custom (carrier-specific) symbol characters and sent as part of text messages, emails, and web pages. In theory, they are confined to each cell phone carrier's network unless there is an agreement and a converter in place between two carriers. In practice, however, people expect emoji just to work - what they put into a message will get to all the recipients; what they see on a web page will be seen by others; if they search for a character they'll find it. For that to really work well, these symbol characters need to be part of the Unicode Standard (the universal character set used in modern computing).

There are active, on-going efforts to standardize a complete set of emoji as regular symbols characters in Unicode. This involves determining which symbols are already covered in Unicode, and which new symbols would be needed. We're trying to help this effort along by sharing all of our mapping data and tools in the form of the "emoji4unicode" open source project. The goal is more effective collaboration with other members of the Unicode Consortium and review by the cell phone carriers and other interested parties. By making these tools and mappings available, we hope to assist and accelerate the encoding process. Take a look at the documentation, browse the data and tools and let us know what you think.

WHOPR – A scalable whole program optimizer for GCC

Jeff Bailey — Thu, 20 Nov 2008 23:31:00 +0000

By Diego Novillo, Google Compiler Team

Traditional compilation proceeds one file at a time. The compiler optimizes and generates code for each file in isolation and then the final executable is created by linking all the individual files together.

This model of compilation has the advantage that all the files can be compiled concurrently, which greatly reduces build time. This is particularly useful with current multiprocessor machines and applications consisting of hundreds and even thousands of files. However, this model presents a fairly significant barrier for optimization.

Consider this program:

foo.c:

foo()
{
   for (;;) {
     ...
     x += g (f (i, j), f (j, i));
     ...
   }
}

bar.c:

float f(float i, float j)
{
  return i * (i - j);
}

float g(float x, float y)
{
  return x - y;
}

From an optimization perspective, inlining f and g inside foo is likely going to provide significant performance improvements. However, the compiler never sees both files at the same time. When it is compiling foo.c it does not have access to the functions in bar.c and vice-versa. Therefore, inlining will never take place.

One could get around this problem by moving the bodies of f and g to a common header file and declaring them inline. But that is not always desirable or even possible. So, several optimizing compilers introduce a new feature called Link-Time Optimization (LTO) that allows these kinds of cross-file manipulations by the compiler.

The LTO model essentially splits code generation in two major phases

1. Generation of Intermediate Representation (IR). The original source code is parsed as usual and an intermediate representation (IR) for the program is generated. The IR is a succinct representation of the original program, symbols and types. This contains all the information that the compiler needs to generate final code, except that instead of using it to generate final code for each file, the compiler saves it to an intermediate file for later processing.

2. Once the IR has been emitted for all the source files, the intermediate files generated in the previous step are loaded in memory and the whole set is analyzed and optimized at once.

To visualize this process, imagine that you simply concatenated all the source files together and compiled the result. Since the compiler has visibility over every function in every compilation unit, decisions that would normally use conservative estimates can instead be based on data-flow information crossing file boundaries. Additionally, the compiler is able to perform cross-language optimizations that are not possible when the compilation scope is restricted to individual files.

The LTO model of compilation is useful but it has severe scalability issues. A basic implementation has the potential to incur massive memory consumption during compilation. Since every function body in every file may be needed in memory, only relatively small programs will be able to be compiled in whole program mode.

At Google, we deal with several massively large applications, so we are working on a scalable alternative to traditional LTO called WHOPR (WHOle Program optimizeR), which introduces parallelism and distribution to be able to handle arbitrarily large programs. The basic observation is that to do many whole program optimizations, the compiler rarely needs to have all the functions loaded in memory, and final code generation can be parallelized by partitioning the program into independent sets.

WHOPR then proceeds in 3 phases

1. Local Generation (LGEN). This is the same as traditional LTO. Every source file is parsed and its IR saved to disk. This phase is trivially parallelizable using make -j or distcc or any other similar technique.

2. Whole program analysis (WPA). After all the IR files are generated, they are sent to the linker, but in this case the linker will not know what to do with them (there is no object code in them). So, the linker turns around and passes them back to the compiler which will collect summary information from every function in every file. This per-function summary information contains things like number of instructions, symbols accessed, functions called, functions that call it, etc. It is used to decide what optimizations to apply, but no optimizations are applied at this time. The compiler simply decides what to do and partitions the input files into new files that contain the original IR plus an optimization plan for each new file.

3. Local transformations (LTRANS). The new IR files generated by the previous phase are now compiled to object code using the optimization plan decided by WPA. Since each file contains everything needed to apply the optimization plan, it can also proceed in parallel.

This diagram shows the process. The only sequential step during optimization is the WPA phase, which does not need to operate on too much data and it is not computationally expensive. Everything else proceeds in parallel, suitable for multiprocessors or distributed machines.

After all the LTRANS processes are finished, the final object files are returned to the linker and the final executable is generated.

We are currently in the initial stages of implementation. The work is being implemented in the LTO branch in GCC. We expect to have an initial prototype by summer 2009. The branch can currently deal with some applications, but there are the usual rough spots. You can read more information about the project at http://gcc.gnu.org/wiki/whopr

Gerrit and Repo, the Android Source Management Tools

Jeff Bailey — Mon, 03 Nov 2008 23:14:00 +0000

By Jeff Bailey, Open Source Team

A couple weeks ago, we announced the Android open source release. Beside it, we silently released the tools that we wrote in order to make handling a large multi-repository project manageable in git. If you had a chance to look through the Android open source website, you'll notice references to a tool called repo. Why did we write this? With approximately 8.5 million lines of code (not including things like the Linux Kernel!), keeping this all in one git tree would've been problematic for a few reasons:

* We want to delineate access control based on location in the tree.
* We want to be able to make some components replaceable at a later date.
* We needed trivial overlays for OEMs and other projects who either aren't ready or aren't able to embrace open source.
* We don't want our most technical people to spend their time as patch monkeys.

The repo tool uses an XML-based manifest file describing where the upstream repositories are, and how to merge them into a single working checkout. repo will recurse across all the git subtrees and handle uploads, pulls, and other needed items. repo has built-in knowledge of topic branches and makes working with them an essential part of the workflow.

The gerrit code review tool is based off of rietveld. Gerrit is itself split into two components: Half that runs on Google App Engine to provide front-end web service, and half that runs on a machine to handle attempted merges into the "upstream" branch, and the various code review branches. When we integrate the auto-builders into the system, that will also be handled by Gerrit.

We have a workflow diagram that shows how code gets into the system for Android. If you're looking to switch to git, but don't want to lose the ability for multiple people to commit into an upstream tree, this is one solution for you to consider. Interested? Find us at repo-discuss@googlegroups.com

First Android Patch Accepted!

Jeff Bailey — Tue, 21 Oct 2008 23:39:00 +0000

By Jeff Bailey, Open Source Team

This morning at 8 AM Pacific, I had the joy of participating in the Android release. If you've been following along, you'll have seen how excited we've been - and are - to publish millions of lines of code to the outside world.

Well, the number just went up by six more lines. It's a small start, but knowing that we accepted our first patch from a contributor external to the Open Handset Alliance just 4.5 hours after unveiling the code reinforces to me why open sourcing this is exactly the right thing to do.

Happy Hacking. =)

(Update: I just checked, and we're up to 5 accepted patches from 8 submitted. Way cool.)

Summer of Coders at Google Krakow

Jeff Bailey — Sun, 31 Aug 2008 06:48:00 +0000

By Martin Harvan, 2008 Student for the SIP Communicator project

I was thrilled when I found out there would be a Google Summer of Code meetup in Krakow, not far from where I live. I not only wanted to visit Google's offices and see behind the scenes of Google, I was excited to meet fellow Summer of Code students and some of Google's engineers. The office visit, which was held on August 4th, turned out to be just as fun as I anticipated.

After we arrived, our hosts showed us around the office and offered "java" and snacks. Following refreshments, a few of the employees informally introduced the projects they are working on and told us about life at Google. It was very interesting to see how many different projects are done in this single office.

Each of us 20+ students then got a short time to introduce what we were working on for our Summer of Code projects. Once again it was interesting to see the wide variety of projects. After we finished talking about our projects, we were given some cool Google swag and there was time for discussion, more food and some fun (table tennis, Wii tennis, RockBand etc.). Time went by very quickly and too soon it was time to go, so we took some pictures and said goodbye to our hosts.

Sojourning in Szeged? Stop by DrupalCon 2008

Jeff Bailey — Thu, 21 Aug 2008 23:49:00 +0000

By Brian W. Fitzpatrick, Open Source Team

If you're passionate about web development and will be in or around Szeged, Hungary next week, be sure to stop by DrupalCon 2008. Nearly 500 developers will be descending on the city to discuss all things Drupal, from improving user experience to further growing this CMS' vibrant community. Our very own Leslie Hawthorn, geek herder extraordinaire, will be presenting on Open Source at Google, with highlights from Drupal's participation in the Google Summer of Code™ program and the Google Highly Open Participation Contest™.

If you can make it, be sure to swing by to say hello to Leslie!

… and the winners of the 2008 Google-O’Reilly Open Source Awards are…

Jeff Bailey — Wed, 23 Jul 2008 02:02:00 +0000

By Jeff Bailey, Open Source Team

Hello from OSCON! This evening, we hosted the fourth annual Google-O'Reilly Open Source Awards. I'm pleased to announce, for those who weren't present, the winners of the 2008 awards:

Best Community Amplifier: Chris Messina - BarCamp, Microformats and Spread Firefox
Best Contributor: Angela Byron - Drupal
Best Education Enabler: Martin Dougiamas - Moodle
Best Interoperator: Andrew Tridgell - Samba and Rsync
Defender of Rights: Harald Welte - gplviolations.org

Awards were given out by Google's Chris DiBona and Parrot's Allison Randall to the various recipients, with Bryan Williams accepting on behalf of Martin.

For a special treat, Leslie Hawthorn got up and spoke about Angela's award.

As always, we're having a good time and many of us will be working the Google booth on the exhibition floor tomorrow and Thursday. I'll hope you'll come say "Hi!"