Google Data » Open Source Programs Office

Open source down under: Linux.conf.au 2017

Open Source Programs Office — Wed, 28 Dec 2016 17:00:00 +0000

It’s a new year and open source enthusiasts from around the globe are preparing to gather at the edge of the world for Linux.conf.au 2017. Among those preparing are Googlers, including some of us from the Open Source Programs Office.

This year Linux.conf.au is returning to Hobart, the riverside capital of Tasmania, home of Australia’s famous Tasmanian devils, running five days between January 16 and 20.

Tuz, a Tasmanian devil sporting a penguin beak, is the Linux.conf.au mascot.
(Artwork by Tania Walker licensed under CC BY-SA.)

The conference, which began in 1999 and is community organized, is well equipped to explore the theme, "the Future of Open Source," which is reflected in the program schedule and miniconfs.

You’ll find Googlers speaking throughout the week (listed below), as well as participating in the hallway track. Don’t miss our Birds of a Feather session if you’re a student, educator, project maintainer, or otherwise interested in talking about outreach and student programs like Google Summer of Code and Google Code-in.

Monday, January 16th
12:20pm The Sound of Silencing by Julien Goodwin
4:35pm Year of the Linux Desktop? by Jessica Frazelle

Tuesday, January 17th
All day Community Leadership Summit X at LCA

Wednesday, January 18th
2:15pm Community Building Beyond the Black Stump by Josh Simmons
4:35pm Contributing to and Maintaining Large Scale Open Source Projects by Jessica Frazelle

Thursday, January 19th
4:35pm Using Python for creating hardware to record FOSS conferences! by Tim Ansell

Friday, January 20th
1:20pm Linux meets Kubernetes by Vishnu Kannan

Not able to make it to the conference? Keynotes and sessions will be livestreamed, and you can always find the session recordings online after the event.

We’ll see you there!

By Josh Simmons, Open Source Programs Office

Taking the pulse of Google Code-in 2016

Open Source Programs Office — Fri, 23 Dec 2016 17:00:00 +0000

Today is the official midpoint of this year’s Google Code-in contest and we are delighted to announce this is our most popular year ever! 930 teenagers from 60 countries have completed 3,503 tasks with 17 open source organizations. The number of students successfully completing tasks has almost met the total number of students from the 2015 contest already.

Tasks that the students have completed include:

writing test suites
improving mobile UI
writing documentation and creating videos to help new users
working on internationalization efforts
fixing and finding bugs in the organization's’ software

Participants from all over the world
In total, over 2,800 students from 87 countries have registered for the contest and we look forward to seeing great work from these (and more!) students over the next few weeks. 2016 has also seen a huge increase in student participation in places such as Indonesia, Vietnam and the Philippines.

Google Code-in participants by country

Please welcome two new countries to the GCI family: Mauritius and Moldova! Mauritius made a very strong debut to the contest and currently has 13 registered students who have completed 31 tasks.

The top five countries with the most completed tasks are:

India: 982
United States: 801
Singapore: 202
Vietnam: 119
Canada: 117

Students, there is still plenty of time to get started with Google Code-in. New tasks are being added daily to the contest site — there are over 1,500 tasks available for students to choose from right now! If you don’t see something that interests you today, check back again every couple of days for new tasks.

The last day to register for the contest and claim a task is Friday, January 13, 2017 with all work being due on Monday, January 16, 2017 at 9:00 am PT.

Good luck to all of the students participating this year in Google Code-in!

By Stephanie Taylor, Google Code-in Program Manager

All numbers reported as of 8:00 PM Pacific Time, December 22, 2016.

Google Summer of Code 2016 wrap-up: Public Lab

Open Source Programs Office — Wed, 21 Dec 2016 17:00:00 +0000

This post is part of our series of guest posts from students, mentors and organization administrators who participated in Google Summer of Code 2016.

How we made this our best Google Summer of Code ever

This was our fourth year doing Google Summer of Code (GSoC), and it was our best year ever by a wide margin! We had five hard-working students who contributed over 17,000 new lines of (very useful) code to our high-priority projects.

Students voluntarily started coding early and hit the ground running, with full development environments and a working knowledge of GitHub Flow-style pull request process. They communicated with one another and provided peer support. They wrote tests. Hundreds of them! They blogged about their work as they went, and chatted with other community members about how to design features.

All of that was amazing, and it was made better by the fact that we were accepting pull requests with new code twice weekly. Tuesdays and Fridays, I went through new submissions, provided feedback, and pulled new code into our master branch, usually publishing it to our production site once a week.

I don't know how other projects do things, but this was very new for us, and it's revolutionized how we work together. In past years, students would work on their forks, slowly building up features. Then in a mad dash at the end, we’d try to merge them into trunk, with lots of conflicts and many hours (weeks!) of work on the part of project maintainers.

What made this year so good?

Many things aligned to make this summer great, and basically none of them are our ideas. I'm sure plenty of you are cringing at how we used to do things, but I also don't think that it's that unusual for projects not "born" in the fast-paced world of modern code collaboration.

We used ideas and learned from Nicolas Bevacqua, author of JavaScript Application Design and of the woofmark and horsey libraries which I've contributed to. We've also learned a great deal from the Hoodie community, particularly Gregor Martynus, who we ran into at a BostonJS meetup. Lastly, we learned from SpinachCon, organized by Shauna Gordon McKeon and Deb Nicholson, where people refine their install process by actually going through the process while sitting next to each other.

Broadly, our strategies were:

Good documentation for newcomers (duh)
Short and sweet install process that you've tried yourself (thanks, SpinachCon!)
Predictable, regular merge schedule
Thorough test suite, and requiring tests with each pull request
Modularity, insisting that projects be broken into small, independently testable parts and merged as they’re written

Installation and pull requests

Most of the above sound kind of obvious or trivial, but we saw a lot of changes when we put it all together. Having a really fast install process, and guidance on getting it running in a completely consistent environment like the virtualized Cloud9 service, meant that many students were able to get the code running the same day they found the project. We aimed for an install time of 15 minutes max, and supplied a video of this for one of our codebases.

We also asked students to make a small change (even just add a space to a file) and walk through the GitHub Flow pull request (PR) submission process. We had clear step-by-step guidance for this, and we took it as a good sign when students were able to read through it and do this.

Importantly, we really tried to make each step welcoming, not demanding or dismissive, of folks who weren’t familiar with this process. This ultimately meant that all five students already knew the PR process when they began coding.

Twice-weekly merge schedule

We were concerned that, in past years, students only tried merging a few times and typically towards the end of the summer. This meant really big conflicts (with each other, often) and frustration.

This year we decided that, even though we’re a tiny organization with just one staff coder, we’d try merging on Tuesday and Friday mornings, and we mostly succeeded. Any code that wasn’t clearly presented, commits squashed, passing tests, and submitting new tests, was reviewed and I left friendly comments and requests so it could be merged the following week.

At first I felt bad rejecting PRs, but we had such great students that they got used to the strictness. They got really good at separating out features, demonstrating their features through clear tests, and some began submitting more than two PRs per week - always rebasing on top of the latest master to ensure a linear commit history. Sweet!

Wrap-up and next steps

The last thing we did was to ask each student, essentially as their documentation, to write a series of new issues which clearly described the problem and/or desired behavior, leave suggestions and links to specific lines of code or example code, and mark them with the special “help-wanted” tag which was so helpful to them when they first started out. We asked each to also make one extra-welcoming “first-timers-only” issue which walks a new contributor through every step of making a commit and even provides suggested code to be inserted.

This final requirement was key. While I personally made each of the initial set of “help-wanted” and “first-timers-only” issues before GSoC, now five students were offloading their unfinished to-dos as very readable and inviting issues for others. The effect was immediate, in part because these special tags are syndicated on some sites. Newcomers began picking them up within hours and our students were very helpful in guiding them through their first contributions to open source.

I want to thank everyone who made this past summer so great, from our champion mentors and community members, to our stellar students, to all our inspirations in this new process, to the dozen or so new contributors we’ve attracted since the end of August.

By Jeff Warren, Organization Administrator for PublicLab.org

Google Summer of Code 2016 wrap-up: CSE@TU Wien

Open Source Programs Office — Wed, 14 Dec 2016 17:00:00 +0000

Every year over a thousand university students work with more than a hundred open source organizations as part of the Google Summer of Code (GSoC). This post is part of a series of guest posts from students, mentors and organization administrators reflecting on GSoC 2016.

CSE@TU Wien is a loose interest group at the Technische Universität Wien (TU Wien) focused on developing, providing and utilizing free and open source software for research. We’re an umbrella organization for several open source projects and we participate in Google Summer of Code (GSoC) to ensure that future generations continue building open source software for scientific computing.

We’ve participated in GSoC most years since 2011, and in 2016 we had ten successful projects. The thematic areas are -- befitting an engineering-focused university -- very diverse. Let’s take a look at the projects and what students accomplished:

Carbon Footprint for Google Maps is a browser extension that calculates CO2 emissions that users would incur by driving on routes suggested by popular mapping services and displays this information alongside time and distance. The aim is to raise awareness of the environmental impact of driving cars.

Kolya Opahle brilliantly re-factored the extension, making it much more modular. This enabled expansion to include other map services and port to other browsers, with browser-specific implementations reduced to a minimum. Building for specific browsers was made easy through a Gradle build script. He took on the Firefox port himself, which turned out to be more challenging than expected due to incompatibilities between the extension API’s of Firefox and Chrome. Overcoming this challenge required ingenuity.

Prateek Gupta completely re-designed and reimplemented the extension’s user interface, optimizing the storage of user options and allowing localization. He added support for more mapping services and calculations of additional greenhouse gases. He added new features to give the user more information about greenhouse gas emissions, including:

a page with air quality index using an API from the World Air Quality Index
a page with tips to reduce emissions; a calculator to compute CO2 absorption by trees
another calculator for the benefits of walking and cycling instead of driving

Chirag Arora ported the extension to the Safari web browser. Like the port to Firefox, this proved challenging due to discrepancies between the Chrome and Safari extension API’s. Chirag also implemented several new features, including:

more unit systems in the options page
automatic configuration of fuel price based on location and the Global Petrol Prices API
approximate calculation of CO2 emissions for public transportation

The Colibri project focuses on smart building energy management. Intelligent control strategies are becoming more and more important for efficiently operating residential and commercial buildings, as buildings are responsible for a significant amount of global energy consumption.

Georg Faustmann implemented a connector for Open Automated Demand Response (OpenADR) networks. OpenADR information and signals can now be processed and stored in the Colibri data store. One challenge for this student was comprehensive handling of the OpenADR specification. Based on the specification, Georg identified a set of relevant use cases which were finally realized in this Colibri component.

Josef Wechselauer worked on a connector for gateways based on the OASIS Open Building Information Exchange (OBIX) standard. This connector links physical devices and data from building automation systems to Colibri. Josef was very enthusiastic and he implemented the connector with an additional graphical user interface for browsing through available OBIX objects. The system test with real hardware was challenging, but he solved all of the problems.

Pratyush Talreja implemented a connector that enables the integration of MATLAB Simulink simulations. More precisely, the connector links to the MATLAB environment and can read and write data over interfaces provided by the simulation. Pratyush had some initial troubles with the system design and the role of the connector in the overall system. However, he tackled those challenges and succeeded in the end.

Mind the Word is a browser extension that helps users learn a new language. It randomly translates a few words per sentence on websites as the user browsers. Since the user sees the translated words in context, they can infer its meaning and thus gradually learns new vocabulary with minimal effort. The extension uses Google, Microsoft and Yandex translation APIs.

Ankit Muchhala re-factored and modernized the code base to ES6 using JSPM, fixing critical bugs in the process and setting up a test environment in Karma and Jasmine. After that, he redesigned the user interface, making extensive use of Bootstrap 3 along with AngularJS. He also implemented various features to make the extension more usable, such as:

dispersed word translation
(automatic) blacklisting and easy whitelisting of words and websites
and the ability to backup and restore the user's configurations

Rohan Katyal ported the extension to Firefox and implemented several new features, including:

speech of translated words
generation of quizzes with the translated words
search for visual hints, similar words and usage examples, and more.

R/sdcMicro is the state-of-the-art R package for data anonymization and is used by national and international institutions. Data privacy has become a hot topic in research and requires serious effort to ensure that individuals cannot be identified.

Probhonjon Baruah improved the code quality of sdcMicro. He wrote unit tests that should help other contributors keep the package consistent and free of bugs. The main challenge for the student was understanding the object-oriented implementation of sdcMicro that goes beyond typical R packages. The student learned that standardized tests are too general to be useful, and that more problem-oriented and specific tests are more effective.

Classilist is an open source visualization dashboard for probabilistic classification data.

Medha Katehara of LNMIIT India developed Classilist, an interactive system for visualizing the performance of probabilistic classifiers. Additionally, she developed plugins to pull classification data from machine learning frameworks such as RapidMiner, WEKA and R.

In conclusion, we are -- again -- very happy with Google Summer of Code. Students advanced themselves and our research software, a clear win-win. Our large team of experienced mentors performed well and we’re grateful for their continued dedication and the support of our university. We hope to participate again in 2017!

By Josef Weinbub and Florian Rudolf, Organization Administrators for TU Wien, Austria

Open sourcing the Embedding Projector: a tool for visualizing high dimensional data

Open Source Programs Office — Mon, 12 Dec 2016 17:00:00 +0000

Originally posted on the Google Research Blog

Recent advances in machine learning (ML) have shown impressive results, with applications ranging from image recognition, language translation, medical diagnosis and more. With the widespread adoption of ML systems, it is increasingly important for research scientists to be able to explore how the data is being interpreted by the models. However, one of the main challenges in exploring this data is that it often has hundreds or even thousands of dimensions, requiring special tools to investigate the space.

To enable a more intuitive exploration process, we are open-sourcing the Embedding Projector, a web application for interactive visualization and analysis of high-dimensional data recently shown as an A.I. Experiment, as part of TensorFlow. We are also releasing a standalone version at projector.tensorflow.org, where users can visualize their high-dimensional data without the need to install and run TensorFlow.

Exploring Embeddings

The data needed to train machine learning systems comes in a form that computers don't immediately understand. To translate the things we understand naturally (e.g. words, sounds, or videos) to a form that the algorithms can process, we use embeddings, a mathematical vector representation that captures different facets (dimensions) of the data. For example, in this language embedding, similar words are mapped to points that are close to each other.

With the Embedding Projector, you can navigate through views of data in either a 2D or a 3D mode, zooming, rotating, and panning using natural click-and-drag gestures. Below is a figure showing the nearest points to the embedding for the word “important” after training a TensorFlow model using the word2vec tutorial. Clicking on any point (which represents the learned embedding for a given word) in this visualization, brings up a list of nearest points and distances, which shows which words the algorithm has learned to be semantically related. This type of interaction represents an important way in which one can explore how an algorithm is performing.

Methods of Dimensionality Reduction

The Embedding Projector offers three commonly used methods of data dimensionality reduction, which allow easier visualization of complex data: PCA, t-SNE and custom linear projections. PCA is often effective at exploring the internal structure of the embeddings, revealing the most influential dimensions in the data. t-SNE, on the other hand, is useful for exploring local neighborhoods and finding clusters, allowing developers to make sure that an embedding preserves the meaning in the data (e.g. in the MNIST dataset, seeing that the same digits are clustered together). Finally, custom linear projections can help discover meaningful "directions" in data sets - such as the distinction between a formal and casual tone in a language generation model - which would allow the design of more adaptable ML systems.

A custom linear projection of the 100 nearest points of "See attachments." onto the "yes" - "yeah" vector (“yes” is right, “yeah” is left) of a corpus of 35k frequently used phrases in emails

The Embedding Projector website includes a few datasets to play with. We’ve also made it easy for users to publish and share their embeddings with others (just click on the “Publish” button on the left pane). It is our hope that the Embedding Projector will be a useful tool to help the research community explore and refine their ML applications, as well as enable anyone to better understand how ML algorithms interpret data. If you'd like to get the full details on the Embedding Projector, you can read the paper here. Have fun exploring the world of embeddings!

By Daniel Smilkov and the Big Picture group

Google Summer of Code 2016 wrap-up: AOSSIE

Open Source Programs Office — Fri, 09 Dec 2016 17:00:00 +0000

We’re sharing guest posts from students, mentors and organization administrators who participated in Google Summer of Code (GSoC) 2016. This is the seventh post in the series.

AOSSIE (Australian Open Source Software Innovation and Education) is an organization created by the leaders of four research-oriented open source projects at the Australian National University. This was our first year in Google Summer of Code, but one of our projects had already participated three times as part of another organization.

We had 6 students and they surpassed our expectations. It was a great experience to mentor these students and provide them the opportunity to get involved in our cutting-edge research. We expect that their projects will lead to several publications and will be the starting point for long term collaborations.

Here are some highlights of their contributions:

Extempore is a programming language and runtime environment that supports live programming.

Joseph Penington adapted some cpp fluid dynamics code to show how live programming could be used to improve the workflow of scientific simulation. Joseph's project builds a series of increasingly complex fluid solvers in Extempore, allowing the programmer to make interesting and non-trivial changes to the simulation at runtime, including switching the way the fluids are solved in the middle of a simulation.

PriMedLink is software for matching similar patients in a way that preserves privacy (i.e. only using masked or encoded values of records without compromising privacy and confidentiality of patients) for health informatics applications such as clinical trials, advanced treatments and personalized patient care. The initial version of PPSPM software included masking and matching techniques for string, categorical and numerical (integer, floating point and modulus) data.

Mathu Mounasamy developed a module for PPSPM for masking and matching textual data which commonly occur in patient records (such as clinical notes and medical reports containing text data). The TextMM module developed by Mathu extends the functionality of PPSPM by allowing advanced privacy-preserving matching of similar patients based on various features containing textual data, thereby improving the quality and scope of PPSPM.

Rogas is a platform which integrates a collection of graph analysis tools and algorithms into a unified framework in order to support network analysis tasks.

Mojtaba Rezvani added the local community search (also known as local community detection) capability to Rogas. He has implemented several state-of-the-art algorithms proposed for local community detection, such as: k-core, k-truss, k-edge-connected, γ-quasi, and k-cliques. He has also designed a new algorithm for local community detection, which can efficiently identify local communities in large-scale networks.

Yan Xiao redesigned the GUI of Rogas in order to improve usability. He also implemented several visualization techniques to support the graph primitives of Rogas, including cluster, rank and path finding. These developments support dynamic network analysis at different scales so as to predict trends and patterns.

Skeptik is a Scala-based framework for proof theory and automated reasoning.

Ezequiel Postan generalized a challenging proof compression algorithm (the Split algorithm) from propositional logic to first-order logic and implemented it. This enables Skeptik to execute this algorithm not only on proofs output by SAT- and SMT-solvers but also on proofs output by resolution-based automated theorem provers. Ezequiel also implemented parsers for the TPTP and TSTP formats for theorem proving problems and proofs, and implemented a random proof generator to allow comprehensive experimental evaluation of the algorithms.

Daniyar Itegulov implemented a theorem prover for classical first-order logic using Skeptik's data structures and based on a novel logical calculus recently proposed by his mentor. This new calculus, called Conflict Resolution, is inspired by the propositional conflict-driven clause learning procedure used by SAT- and SMT-solvers and generalizes it to first-order logic. Daniyar also went further, conceiving and developing a concurrent proof search strategy for this calculus using Akka actors.

By Bruno Paleo, Organization Administrator for AOSSIE

Open-sourcing DeepMind Lab

Open Source Programs Office — Wed, 07 Dec 2016 17:00:00 +0000

Originally posted on DeepMind Blog

DeepMind's scientific mission is to push the boundaries of AI, developing systems that can learn to solve any complex problem without needing to be taught how. To achieve this, we work from the premise that AI needs to be general. Agents should operate across a wide range of tasks and be able to automatically adapt to changing circumstances. That is, they should not be pre-programmed, but rather, able to learn automatically from their raw inputs and reward signals from the environment. There are two parts to this research program: (1) designing ever-more intelligent agents capable of more-and-more sophisticated cognitive skills, and (2) building increasingly complex environments where agents can be trained and evaluated.

The development of innovative agents goes hand in hand with the careful design and implementation of rationally selected, flexible and well-maintained environments. To that end, we at DeepMind have invested considerable effort toward building rich simulated environments to serve as “laboratories” for AI research. Now we are open-sourcing our flagship platform, DeepMind Lab, so the broader research community can make use of it.

DeepMind Lab is a fully 3D game-like platform tailored for agent-based AI research. It is observed from a first-person viewpoint, through the eyes of the simulated agent. Scenes are rendered with rich science fiction-style visuals. The available actions allow agents to look around and move in 3D. The agent’s “body” is a floating orb. It levitates and moves by activating thrusters opposite its desired direction of movement, and it has a camera that moves around the main sphere as a ball-in-socket joint tracking the rotational look actions. Example tasks include collecting fruit, navigating in mazes, traversing dangerous passages while avoiding falling off cliffs, bouncing through space using launch pads to move between platforms, playing laser tag, and quickly learning and remembering random procedurally generated environments. An illustration of how agents in DeepMind Lab perceive and interact with the world can be seen below:

At each moment in time, agents observe the world as an image, in pixels, rendered from their own first-person perspective. They also may receive a reward (or punishment!) signal. The agent can activate its thrusters to move in 3D and can also rotate its viewpoint along both horizontal and vertical axes.

Artificial general intelligence research in DeepMind Lab emphasizes navigation, memory, 3D vision from a first person viewpoint, motor control, planning, strategy, time, and fully autonomous agents that must learn for themselves what tasks to perform by exploring their environment. All these factors make learning difficult. Each are considered frontier research questions in their own right. Putting them all together in one platform, as we have, represents a significant new challenge for the field.

DeepMind Lab is highly customisable and extendable. New levels can be authored with off-the-shelf editor tools. In addition, DeepMind Lab includes an interface for programmatic level-creation. Levels can be customised with gameplay logic, item pickups, custom observations, level restarts, reward schemes, in-game messages and more. The interface can be used to create levels in which novel map layouts are generated on the fly while an agent trains. These features are useful in, for example, testing how an agent copes with unfamiliar environments. Users will be able to add custom levels to the platform via GitHub. The assets will be hosted on GitHub alongside all the code, maps and level scripts. Our hope is that the community will help us shape and develop the platform going forward.

DeepMind Lab has been used internally at DeepMind for some time (example). We believe it has already had a significant impact on our thinking concerning numerous aspects of intelligence, both natural and artificial. However, our efforts so far have only barely scratched the surface of what is possible in DeepMind Lab. There are opportunities for significant contributions still to be made in a number of mostly still untouched research domains now available through DeepMind Lab, such as navigation, memory and exploration.

As well as facilitating agent evaluation, there are compelling reasons to think that it may be fundamentally easier to develop intelligence in a 3D world, observed from a first-person viewpoint, like DeepMind Lab. After all, the only known examples of general-purpose intelligence in the natural world arose from a combination of evolution, development, and learning, grounded in physics and the sensory apparatus of animals. It is possible that a large fraction of animal and human intelligence is a direct consequence of the richness of our environment, and unlikely to arise without it. Consider the alternative: if you or I had grown up in a world that looked like Space Invaders or Pac-Man, it doesn’t seem likely we would have achieved much general intelligence!

Read the full paper here.

Access DeepMind's GitHub repository here.

By Charlie Beattie, Joel Leibo, Stig Petersen and Shane Legg, DeepMind Team

Why I contribute to Chromium

Open Source Programs Office — Mon, 05 Dec 2016 17:00:00 +0000

This is a guest post by Yoav Weiss who was recently recognized through the Google Open Source Peer Bonus Program for his work on the Chromium project. We invited Yoav to share about his work on our blog.

I was recently recognized by Google for my contributions to Chromium and wanted to write a few words on why I contribute to the project, other rendering engines and the web platform in general. I also wanted to share how it helped me evolve as a developer and why more people should contribute to the web platform for their own benefit.

The web platform

I’ve written before about why I think the web platform is an extremely important asset for humanity and why we should make sure it'll thrive for years to come. It enables the distribution of knowledge to the corners of the earth and has fundamentally changed our world. Yet, compared to the amount of users (billions!) and web developers (millions), there are only a few hundred engineers working on maintaining and improving the platform itself.

That means that there are many aspects of the platform that are not as well maintained as they should be. We're at a real risk of a "tragedy of the commons" scenario, where despite usage and utility, the platform will collapse under its own weight because maintaining it is nobody's exclusive problem.

How I got started

Personally, I had been working on web performance for well over a decade before I decided to get more involved and lend my hand in building the platform. For a large part of my professional life, browsers were black boxes. They were given to us by the browser gods and that's what we had to work with for the next few years. Their undocumented bugs and quirks became gospel, passed from senior engineers to their juniors.

Then at some point, that situation changed. Slowly but surely, open source browsers started picking up market share. No longer black boxes, we can actually see what happens on the inside!

I first got involved by joining the responsive images discussions and the Responsive Images Community Group. Then, I saw a tweet from RICG's chair calling to develop a prototype of the current proposal to prove its feasibility and value. And I jumped in.

I created a prototype using Chromium and WebKit, demoed it to anyone that was interested, worked on the proposals and argued the viability of the proposals' approach on the various mailing lists. Eventually, we were able to get some browser folks on board, improve the proposals and their fit to the rest of the platform, and I started working on an implementation.

The amount of work this required was larger than I expected. Eventually I managed to ship the feature in Blink and Chromium, and complete large parts of the implementation in WebKit as well. WOOT!

Success! Now what?

After that project was done, I started looking into what I should do next. I was determined to continue working on browsers and find a gig that would let me do that. So I searched for an employer with a vested interest in the web and in making it faster, who would be happy to let me work on the platform's client - the web browser.

I found such an employer in Akamai, where I have been working as a Principal Architect ever since. As part of my job I'm working on our performance optimization features as well as performance-related browser features, making sure they make it into browsers in a timely fashion.

Why you should contribute, too

Now, chances are that if you're reading this, you're also relying on the web platform for your job in one way or another. Which means that there's a chance that it also makes sense for your organization to contribute to the web platform. Let’s explore the reasons:

1. Make sure work is done on features you care about

If you're like me, you love the web platform and the reach it provides you, but you're not necessarily happy with all of it. The web is great, but not perfect. Since browsers and web standards are no longer black boxes, you can help change that.

You can work on standards and browsers to change them to include your use-case. That's immense power at your fingertips: put in the work and the platform evolves for all the billions of users out there.

And you don’t have to wait years before new features can be used in production like with yesteryear's browser changes. With today’s browser update rates and progressive enhancement, you’ll probably be able to use changes in production within a few months.

2. Gain expertise that can help you do your job better

Knowing browser internals better can also give you superpowers in other parts of your job. Whenever questions about browser behavior arrive, you can take a peek into the source code and have concrete answers rather than speculation.

Keeping track of standards discussions give you visibility into new browser APIs that are coming along, so that you can opt to use those rather than settle for sub-optimal alternatives that are currently available.

3. Grow as an engineer

Working on browsers teaches you a lot about how things work under the surface and enables you to understand the internals of modern browsers, which are extremely complex machines. Further, this work allows you to get code reviews from the world's leading experts on these subjects. What better way to grow than to interact with the experts?

4. It's a fun and welcoming community

Contributing to the web platform has been a great experience for me. Working with the Chromium project, in particular, is always great fun. The project is Google backed, but there are many external contributors and the majority of work and decisions are being done in the open. The people I've worked with are super friendly and happy to help. All in all, it's really fun!

Join us

The web needs more people working on it, and working on the web platform can be extremely beneficial to you, your career and your business.

If you're interested in getting started with web standards, the Discourse instance of the web Platform Incubator Community Group (or WICG for short) is where it's at (disclaimer: I'm co-chairing that group). For getting started with Chromium development, this is the post for you.

And most important, don't be afraid to ask the community. People on blink-dev and IRC are super friendly and will be happy to point you in the right direction.

So come on over and join the good cause. We'll be happy to have you!

By Yoav Weiss, Chromium contributor

Announcing OSS-Fuzz: Continuous fuzzing for open source software

Open Source Programs Office — Thu, 01 Dec 2016 17:00:00 +0000

We are happy to announce OSS-Fuzz, a new Beta program developed over the past years with the Core Infrastructure Initiative community. This program will provide continuous fuzzing for select core open source software.

Open source software is the backbone of the many apps, sites, services, and networked things that make up “the internet.” It is important that the open source foundation be stable, secure, and reliable, as cracks and weaknesses impact all who build on it.

Recent security stories confirm that errors like buffer overflow and use-after-free can have serious, widespread consequences when they occur in critical open source software. These errors are not only serious, but notoriously difficult to find via routine code audits, even for experienced developers. That's where fuzz testing comes in. By generating random inputs to a given program, fuzzing triggers and helps uncover errors quickly and thoroughly.

In recent years, several efficient general purpose fuzzing engines have been implemented (e.g. AFL and libFuzzer), and we use them to fuzz various components of the Chrome browser. These fuzzers, when combined with Sanitizers, can help find security vulnerabilities (e.g. buffer overflows, use-after-free, bad casts, integer overflows, etc), stability bugs (e.g. null dereferences, memory leaks, out-of-memory, assertion failures, etc) and sometimes even logical bugs.

OSS-Fuzz’s goal is to make common software infrastructure more secure and stable by combining modern fuzzing techniques with scalable distributed execution. OSS-Fuzz combines various fuzzing engines (initially, libFuzzer) with Sanitizers (initially, AddressSanitizer) and provides a massive distributed execution environment powered by ClusterFuzz.

Early successes

Our initial trials with OSS-Fuzz have had good results. An example is the FreeType library, which is used on over a billion devices to display text (and which might even be rendering the characters you are reading now). It is important for FreeType to be stable and secure in an age when fonts are loaded over the Internet. Werner Lemberg, one of the FreeType developers, was an early adopter of OSS-Fuzz. Recently the FreeType fuzzer found a new heap buffer overflow only a few hours after the source change:

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x615000000ffa
READ of size 2 at 0x615000000ffa thread T0
SCARINESS: 24 (2-byte-read-heap-buffer-overflow-far-from-bounds)
   #0 0x885e06 in tt_face_vary_cvtsrc/truetype/ttgxvar.c:1556:31

OSS-Fuzz automatically notified the maintainer, who fixed the bug; then OSS-Fuzz automatically confirmed the fix. All in one day! You can see the full list of fixed and disclosed bugs found by OSS-Fuzz so far.

Contributions and feedback are welcome

OSS-Fuzz has already found 150 bugs in several widely used open source projects (and churns ~4 trillion test cases a week). With your help, we can make fuzzing a standard part of open source development, and work with the broader community of developers and security testers to ensure that bugs in critical open source applications, libraries, and APIs are discovered and fixed. We believe that this approach to automated security testing will result in real improvements to the security and stability of open source software.

OSS-Fuzz is launching in Beta right now, and will be accepting suggestions for candidate open source projects. In order for a project to be accepted to OSS-Fuzz, it needs to have a large user base and/or be critical to Global IT infrastructure, a general heuristic that we are intentionally leaving open to interpretation at this early stage. See more details and instructions on how to apply here.

Once a project is signed up for OSS-Fuzz, it is automatically subject to the 90-day disclosure deadline for newly reported bugs in our tracker (see details here). This matches industry’s best practices and improves end-user security and stability by getting patches to users faster.

Help us ensure this program is truly serving the open source community and the internet which relies on this critical software, contribute and leave your feedback on GitHub.

By Mike Aizatsky, Kostya Serebryany (Software Engineers, Dynamic Tools); Oliver Chang, Abhishek Arya (Security Engineers, Google Chrome); and Meredith Whittaker (Open Research Lead).

Docker + Dataflow = happier workflows

Open Source Programs Office — Wed, 30 Nov 2016 17:00:00 +0000

When I first saw the Google Cloud Dataflow monitoring UI -- with its visual flow execution graph that updates as your job runs, and convenient links to the log messages -- the idea came to me. What if I could take that UI, and use it for something it was never built for? Could it be connected with open source projects aimed at promoting reproducible scientific analysis, like Common Workflow Language (CWL) or Workflow Definition Language (WDL)?

Screenshot of a Dockerflow workflow for DNA sequence analysis.

In scientific computing, it’s really common to submit jobs to a local high-performance computing (HPC) cluster. There are tools to do that in the cloud, like Elasticluster and Starcluster. They replicate the local way of doing things, which means they require a bunch of infrastructure setup and management that the university IT department would otherwise do. Even after you’re set up, you still have to ssh into the cluster to do anything. And then there are a million different choices for workflow managers, each unsatisfactory in its own special way.

By day, I’m a product manager. I hadn’t done any serious coding in a few years. But I figured it shouldn’t be that hard to create a proof-of-concept, just to show that the Apache Beam API that Dataflow implements can be used for running scientific workflows. Now, Dataflow was created for a different purpose, namely, to support scalable data-parallel processing, like transforming giant data sets, or computing summary statistics, or indexing web pages. To use Dataflow for scientific workflows would require wrapping up shell steps that launch VMs, run some code, and shuttle data back and forth from an object store. It should be easy, right?

It wasn’t so bad. Over the weekend, I downloaded the Dataflow SDK, ran the wordcount examples, and started modifying. I had a “Hello, world” proof-of-concept in a day.

To really run scientific workflows would require more, of course. Varying VM shapes, a way to pass parameters from one step to the next, graph definition, scattering and gathering, retries. So I shifted into prototyping mode.

I created a new GitHub project called Dockerflow. With Dockerflow, workflows can be defined in YAML files. They can also be written in pretty compact Java code. You can run a batch of workflows at once by providing a CSV file with one row per workflow to define the parameters.

Dataflow and Docker complement each other nicely:

Dataflow provides a fully managed service with a nice monitoring interface, retries, graph optimization and other niceties.
Docker provides portability of the tools themselves, and there's a large library of packaged tools already available as Docker images.

While Dockerflow supports a simple YAML workflow definition, a similar approach could be taken to implement a runner for one of the open standards like CWL or WDL.

To get a sense of working with Dockerflow, here’s “Hello, World” written in YAML:

defn:
  name: HelloWorkflow
steps:
- defn:
    name: Hello
    inputParameters:
      name: message
      defaultValue: Hello, World!
    docker:
      imageName: ubuntu
      cmd: echo $message

And here’s the same example written in Java:

public class HelloWorkflow implements WorkflowDefn {
  @Override
  public Workflow createWorkflow(String[] args) throws IOException {
    Task hello =
        TaskBuilder.named("Hello").input("message", “Hello, World!”).docker(“ubuntu”).script("echo $message").build();
    return TaskBuilder.named("HelloWorkflow").steps(hello).args(args).build();
  }
}

Dockerflow is just a prototype at this stage, though it can run real workflows and includes many nice features, like dry runs, resuming failed runs from mid-workflow, and, of course, the nice UI. It uses Cloud Dataflow in a way that was never intended -- to run scientific batch workflows rather than large-scale data-parallel workloads. I wish I’d written it in Python rather than Java. The Dataflow Python SDK wasn’t quite as mature when I started.

Which is all to say, it’s been a great 20% project, and the future really depends on whether it solves a problem people have, and if others are interested in improving on it. We welcome your contributions and comments! How do you run and monitor scientific workflows today?

By Jonathan Bingham, Google Genomics and Verily Life Sciences

Google Summer of Code 2016 wrap-up: STE||AR

Open Source Programs Office — Tue, 29 Nov 2016 17:00:00 +0000

This is part of a series of guest posts from students, mentors and organization administrators who participated in Google Summer of Code (GSoC) 2016. GSoC is an annual program which pairs university students with mentors to work on open source software.

This summer the STE||AR Group was proud to mentor four students through Google Summer of Code. These students worked on a variety of projects which helped improve our software, HPX. This library is a distributed C++ runtime system which supports a standards compliant API and helps users scale their applications across thousands of machines.

The improvements to the code base will help our team and users of HPX around the world. A summary of our students’ projects:

Parsa Amini – HPX Debugger

Developing a better distributed debugging tool is essential to increase the programmability of HPX. Parsa’s project, Scimitar, aims to facilitate the debugging process for HPX programmers by extending the features of GDB, an existing debugger. The project then complements it with new commands for easier switching between localities across clusters, HPX thread debugging, awareness of internal HPX data structures, and semi-automated preparation for distributed debugging sessions. Additional functionality such as locating an object and viewing the queue information on each core is provided through using API provided by HPX itself. His work can be found on GitHub.

Aalekh Nigam – Implement a Map/Reduce Framework

This project aimed to expose a Map/Reduce programming model over HPX. During the summer, Aalekh was able to develop a single node implementation of HPXflow (map/reduce programming model) and laid the groundwork for the further multi-node version with database support. Although the initial task was limited to implementing the Map/Reduce model, he was also able to implement an improved dataflow model as well.

Minh-Khanh Do - Working on Parallel Algorithms for HPX::Vector

Minh-Khanh’s task was to take the parallel algorithms and add the functionality required to work on the segmented hpx::vector. Under his mentor John Biddscombe, he implemented the segmented_fill algorithm, which was successfully merged into the main codebase. Additionally, Minh-Khanh implemented the segmented_scan algorithm which includes inclusive and exclusive_scan. These changes are included in a pull request and have been merged. Using the segmented scan algorithm it is possible to perform tasks such as evaluating polynomials and to implement other algorithms such as quicksort.

Satyaki Upadhyay - Plugin Mechanism for thread schedulers in HPX

In HPX, schedulers are statically linked and must be built at compile-time. Satyaki’s project involved converting this statically linked scheme into a plugin system which would allow arbitrary schedulers to be dynamically loaded. These changes bring several benefits. They provide a layer of abstraction and follow the open/closed principle of software design as well as allowing developers to write their own custom schedulers while conforming to a uniform API. The project proceeded in two steps. The first involved the creation of plugin modules of the schedulers and registering them with HPX. The second step was to implement the loading and subsequent use of the chosen scheduler.

We would like to thank our students and mentors for the time that they have contributed to HPX this summer. In addition, we would like to thank Google for the opportunity that they provided the STE||AR Group to work with developers around the globe as well as the ability for students to interact with vibrant open source projects worldwide.

By Adrian Serio, Organization Administrator for The STE||AR Group

It’s that time again: Google Code-in starts today!

Open Source Programs Office — Mon, 28 Nov 2016 17:15:00 +0000

Today marks the start of the 7th year of Google Code-in (GCI), our pre-university contest introducing students to open source development. GCI takes place entirely online and is open to students between the ages of 13 and 17 around the globe.

The concept is simple: complete bite-sized tasks (at your own pace) created by 17 participating open source organizations on topic areas you find interesting:

Coding
Documentation/Training
Outreach/Research
Quality Assurance
User Interface

Tasks take an average of 3-5 hours to complete and include the guidance of a mentor to help along the way. Complete one task? Get a digital certificate. Three tasks? Get a sweet Google t-shirt. Finalists get a hoodie. Grand Prize winners get a trip to Google headquarters in California.

Over the last 6 years, 3213 students from 99 countries have successfully completed tasks in GCI. Intrigued? Learn more about GCI by checking out our rules and FAQs. And please visit our contest site and read the Getting Started Guide.

Teachers, if you are interested in getting your students involved in Google Code-in you can find resources here to help you get started.

By Mary Radomile, Open Source Programs Office

Stories from Google Code-in: Sugar Labs and Systers

Open Source Programs Office — Wed, 23 Nov 2016 17:00:00 +0000

Google Code-in (GCI) is our annual contest that gives students age 13 to 17 experience in computer science through contributions to open source projects. This blog post is the final installment in our series reflecting on the experiences of Google Code-in 2015 grand prize winners. Be sure to check out the first three posts.

The Google Code-in contest begins on Monday, November 28th at 9am PT for students. Right now you can learn more about the 17 mentoring organizations that students will be able to work with by going to the contest site. To get students excited for GCI 2016, we’re sharing three more stories from GCI 2015 grand prize winners. These stories illustrate how global the competition is, the challenges students face and the valuable skills they learn working with these open source organizations.

A group of Google Code-in 2015 mentors joined grand prize winners for a day of exploring
San Francisco including the iconic Golden Gate Bridge.

First up is the story of Ezequiel Pereira, a student from Uruguay who worked with Sugar Labs. Sugar Labs is the organization behind Sugar, the operating system for the OLPC XO-1 which the Uruguayan government has distributed to public primary schools. The XO-1 was Ezequiel’s first computer.

Ezequiel’s curiosity in computer science was piqued when a technician came to his school to solve a simple bug that was affecting most XO’s. The technician used the command line which, up to that point, Ezequiel thought was useless. Realizing that the command line offered him a lot of power, Ezequiel began his exploration.

He discovered Google Code-in by reading about another Uruguayan teenager, one who was a grand prize winner in Google Code-in 2012. Ezequiel jumped into the contest and participated for several years expanding his skills before finishing as a grand prize winner of Google Code-in 2015. Along the way Ezequiel got comfortable with IRC and began helping other students, even finding new friends among along the way.

Next we have Sara Du from the United States. Sara had been coding for six months when she discovered Google Code-in on Christmas Eve, halfway through the competition. She found lots of interesting tasks, but had trouble finding the right organization to focus on before selecting Systers.

Like many students, Sara was able to quickly jump into code but spent a couple days just getting acquainted with Git and GitHub. This is something we hear from a lot of students and it’s just one of the skills that they pick up by working on real-world projects, along with testing and communication.

Another challenge Sara faced was working with a mentor 16 time zones away from her, which meant that correspondence would often take a day or two. While this was a challenge, she found the long feedback loop encouraged her to get on the Slack channel and reach out to other contributors for help. Ultimately, this made her even more a part of the Systers community.

Sara said Google Code-in was one of the most awesome experiences she’s had and has this advice to offer future participants: “The organization you end up working with has a vibrant community of hackers from everywhere; try to interact with them and you will be sure to learn from others as they will from you!”

Last, but certainly not least, we have Ahmed Sabie, a student from Canada who also worked with Systers. Ahmed started coding competitively several years ago, focusing on graph theory, dynamic programming and data structures. He loved the problem solving, but knew that these competitions took place in a sandbox. To grow, Ahmed would need to explore.

Enter Google Code-in. Ahmed was most comfortable with Python and saw that the Systers Volunteer Management System used that language, so that’s where he started.

Ahmed, like many students and even professional developers, spent much of his first week setting up his development environment. It was a grueling process but with the help of search and the people in the Systers Slack channel he was finally able to see the project’s login screen.

As he completed easy tasks, Ahmed moved on to more difficult tasks and began to help other students, many who got stuck on the same issues he had encountered earlier. Ahmed found that each task provided an opportunity to stretch his skills a little bit more. He was excited about how quickly he was learning. Though Ahmed learned a lot on his own, he says the vast majority of what he learned was through the help of other people -- students, mentors and other project contributors -- and that he felt like he was truly a part of the Systers community by the end of the process.

Ahmed’s favorite task was an appropriate finale for the competition: he added multilingual support to an application he had worked on and added the French translation.

“Overall, Google Code-in was the experience of a lifetime. It set me up for the future, by teaching me relevant and critical skills necessary in software development. I have contributed to a good cause, and met fantastic mentors and friends along the way. Open source development is not a onetime thing, it is an ongoing process. I hope to continue to be part of it, and to me it is a form of volunteering and giving back to the community.” - Ahmed Sabie

With that, we conclude our series of posts reflecting on Google Code-in 2015. We thank Ezequiel, Sara, Ahmed and all the other participants for sharing their stories and contributing to the software we all rely on. We hope you will join us in carrying on the tradition with Google Code-in 2016!

By Josh Simmons, Open Source Programs Office

Google Summer of Code 2016 wrap-up: Linux XIA

Open Source Programs Office — Fri, 18 Nov 2016 18:00:00 +0000

We're sharing guest posts from students, mentors and organization administrators who participated in Google Summer of Code 2016. This is the fifth post in that series and there are more on the way.

Linux XIA is the native implementation of XIA, a meta network architecture that supports evolution of all of its components, which we call “principals,” and promotes interoperability between these principals. It is the second year that our organization, Boston University / XIA, has participated in Google Summer of Code (GSoC), and this year we received 31 proposals from 8 countries.

Our ideas list this year focused on upgrading key forwarding data structures to their best known versions. Our group chose the most deserving students for each of the following projects:

Accelerating the forwarding speed of the LPM principal with poptrie

Student André Ferreira Eleuterio and mentor Cody Doucette implemented the first version of the LPM principal in Linux XIA for GSoC 2015. The LPM principal enables Linux XIA to leverage routing tables derived from BGP, OSPF, IS-IS and any other IP routing protocol to forward XIA packets natively, that is, without encapsulation in IP. For GSoC 2016, student Vaibhav Raj Gupta from India partnered with mentor Cody Doucette to speed up the LPM principal by employing a state-of-the-art data structure to find the longest prefix matching using general purpose processors: poptrie.

Upgrading the FIB hash table of principals to the relativistic hash table

Principals that rely on routing flat names have used a resizable hash table that supports lockless readers since 2011. While this data structure was unique in 2011, in the same year, relativistic hash tables were published. The appeal to upgrade to relativistic hash tables was twofold: reduced memory footprint per hashed element, and the fact they were implemented in the Linux kernel in 2014. Student Sachin Paryani, also from India, worked with mentor Qiaobin Fu to replace our resizable hash table with the relativistic hash table.

Google Summer of Code nurtures a brighter future. Thanks to GSoC, our project has received important code contributions, and our community has been enlarged. It was rewarding to learn that two of our GSoC students have decided to pursue graduate school after their GSoC experience with us: Pranav Goswami (2015) and Sachin Paryani (2016). We hope these examples will motivate other students to do their best because the world is what we make of it.

By Michel Machado, Boston University / XIA organization administrator

Google Summer of Code 2016 wrap-up: Debian

Open Source Programs Office — Wed, 16 Nov 2016 17:30:00 +0000

This is the fourth post in our series of wrap-ups and guest posts from participants reflecting on Google Summer of Code (GSoC) 2016. Explore the first three posts and stay tuned for more wrap-ups and announcements.

Debian, founded in 1993, is a project aimed at building a 100% free and open source “Universal Operating System.” It’s a volunteer-driven project based on Linux, FreeBSD
and Hurd kernels for devices ranging from mobile phones to large clusters.

Being a wide umbrella project, Debian offered a diverse array of opportunities for Google Summer of Code (GSoC) students. For example, students worked on making our distribution more trustworthy (reproducible builds), porting our OS to Android devices and improving infrastructure for developers. This year I joined the Debian Real-Time Communications (RTC) mentoring team which engaged 13 students to improve voice, video and chat communication with free software.

WebRTC, an open standard enabling real-time video and audio communication in the browser, is central to this work. It was used to create JSCommunicator, an embeddable WebRTC phone developed in HTML, CSS and JavaScript, supporting voice, video and chat using SIP over WebSockets. A GSoC 2014 student, Juliana Louback, significantly enhanced JSCommunicator during her summer with Debian.

JSCommunicator is now being adapted for use with content management systems (CMS) and blogging platforms, making it easy to embed rich communication features in existing systems. It was this work that our current GSoC students built on.

This year I mentored GSoC student Mesut Can Gurle who used DruCall, a Drupal module for integrating JSCommunicator, as inspiration for building WPCall for WordPress. With this new plug-in, standards-based voice, video and chat is now available on the world’s two most popular CMS without the need for browser plugins.

The way WPCall was extrapolated from the DruCall plugin provides a pattern that other communities can follow to rapidly create WebRTC plugins for their own web frameworks. The JSCommunicator Integration Guide provides step-by-step instructions that developers and future students can follow. If you’re interested in learning more about significant developments in this space, please subscribe to the Free-RTC Announce mailing list and follow planet.freertc.org.

This was my first year as a GSoC mentor and I had such a great experience. It was rewarding working with Mesut on achieving his goals and we learned a lot along the way. Despite some setbacks (he narrowly missed a bombing as his country experienced an attempted coup), Mesut has made valuable contributions to free software.

As the summer wound down, I received an invitation to participate in a t-shirt design contest for the annual Mentor Summit. I thought it would be fun to try and put together a design focusing on GSoC’s key values.

The front of the t-shirt shows developers from all over the world collaborating on free software, representing the amazing scope and diversity of the projects. On the back, above the clouds, a space shuttle symbolizes what’s achieved through GSoC.

A group of attendees wearing the Google Summer of Code 2016 Mentor Summit t-shirt.

Happily, my design was selected and it was great seeing all the attendees wearing it at the Mentor Summit!

By Bruno Magalhães, Mentor for Debian

ETC2Comp: fast texture compression for games and VR

Open Source Programs Office — Mon, 14 Nov 2016 18:00:00 +0000

For mobile game and VR developers the ETC2 texture format has become an increasingly valuable tool for texture compression. It produces good on-GPU sizes (it stays compressed in memory) and higher quality textures (compared to its ETC1 counterpart).

These benefits come with a significant downside, however: ETC2 textures take significantly longer to compress than their ETC1 counterparts. As adoption of the ETC2 format increases in a project, so do build times. As such, developers have had to make the classic choice between quality and time.

We wanted to eliminate the need for developers to make that choice, so we’ve released ETC2Comp, a fast and high quality ETC2 encoder for games and VR developers.

ETC2 takes a long time to compress textures because the format defines a large number of possible combinations for encoding a block in the texture. To find the most perfect, highest quality compressed image means brute-forcing this incredibly large number of combinations, which clearly is not a time efficient option.

We designed ETC2Comp to get the same visual results at much faster speeds by deploying a few optimization techniques:

Directed Block Search. Rather than a brute-force search, ETC2Comp uses a much more limited, targeted search for the best encoding for a given block. ETC2Comp comes with a precomputed set of archetype blocks, where each archetype is associated with a sorted list of the ETC2 block format types that provide its best encodings. During the actual compression of a texture, each block is initially assigned an archetype, and multiple passes are done to test the block against its block format list to find the best encoding. As a result, the best option can be found much quicker than with a brute-force method.

Full effort setting. During each pass of the encoding process, all the blocks of the image are sorted by their visual quality (worst-looking to best-looking). ETC2Comp takes an effort parameter whose value specifies what percentage of the blocks to update during each pass of encoding. An effort value of 25, for instance, means that on each pass, only the 25% worst looking blocks are tested against the next format in their archetypes' format-chains. The result is a tradeoff between optimizing blocks that already look good, and the time it takes to do it.

Highly multi-threaded code. Since blocks can be evaluated independently during each pass, it’s straightforward to apply multithreading to the work. During encoding ETC2comp can take advantage of available parallel threads, and it even accepts a jobs parameter, where you can define exactly the number of threads you’d like it to use... in case you have a 256 core machine.

Check out the code on GitHub to get started with ETC2Comp and let us know what you think. You can use the tool from the command line or embed the C++ library in your project. If you want to know more about what’s going on under the hood, check out this blog post.

By Colt McAnlis, Developer Advocate

Open source visualization of GPS displacements for earthquake cycle physics

Open Source Programs Office — Thu, 10 Nov 2016 18:00:00 +0000

The Earth’s surface is moving, ever so slightly, all the time. This slow, small, but persistent movement of the Earth's crust is responsible for the formation of mountain ranges, sudden earthquakes, and even the positions of the continents. Scientists around the world measure these almost imperceptible movements using arrays of Global Navigation Satellite System (GNSS) receivers to better understand all phases of an earthquake cycle—both how the surface responds after an earthquake, and the storage of strain energy between earthquakes.

To help researchers explore this data and better understand the Earthquake cycle, we are releasing a new, interactive data visualization which draws geodetic velocity lines on top of a relief map by amplifying position estimates relative to their true positions. Unlike existing approaches, which focus on small time slices or individual stations, our visualization can show all the data for a whole array of stations at once. Open sourced under an Apache 2 license, and available on GitHub, this visualization technique is a collaboration between Harvard’s Department of Earth and Planetary Sciences and Google's Machine Perception and Big Picture teams.

Our approach helps scientists quickly assess deformations across all phases of the earthquake cycle—both during earthquakes (coseismic) and the time between (interseismic). For example, we can see azimuth (direction) reversals of stations as they relate to topographic structures and active faults. Digging into these movements will help scientists vet their models and their data, both of which are crucial for developing accurate computer representations that may help predict future earthquakes.

Classical approaches to visualizing these data have fallen into two general categories: 1) a map view of velocity/displacement vectors over a fixed time interval and 2) time versus position plots of each GNSS component (longitude, latitude and altitude).

Examples of classical approaches. On the left is a map view showing average velocity vectors over the period from 1997 to 2001[1]. On the right you can see a time versus eastward (longitudinal) position plot for a single station.

Each of these approaches have proved to be informative ways to understand the spatial distribution of crustal movements and the time evolution of solid earth deformation. However, because geodetic shifts happen in almost imperceptible distances (mm) and over long timescales, both approaches can only show a small subset of the data at any time—a condensed average velocity per station, or a detailed view of a single station, respectively. Our visualization enables a scientist to see all the data at once, then interactively drill down to a specific subset of interest.

Our visualization approach is straightforward; by magnifying the daily longitude and latitude position changes, we show tracks of the evolution of the position of each station. These magnified position tracks are shown as trails on top of a shaded relief topography to provide a sense of position evolution in geographic context.

To see how it works in practice, let’s step through an an example. Consider this tiny set of longitude/latitude pairs for a single GNSS station, with the differing digits shown in bold:

Day Index	Longitude	Latitude
0	139.06990407	34.949757897
1	139.06990400	34.949757882
2	139.06990413	34.949757941
3	139.06990409	34.949757921
4	139.06990413	34.949757904

If we were to draw line segments between these points directly on a map, they’d be much too small to see at any reasonable scale. So we take these minute differences and multiply them by a user-controlled scaling factor. By default this factor is 10^5.5 (about 316,000x).

To help the user identify which end is the start of the line, we give the start and end points different colors and interpolate between them. Blue and red are the default colors, but they’re user-configurable. Although day-to-day movement of stations may seem erratic, by using this method, one can make out a general trend in the relative motion of a station.

Close-up of a single station’s movement during the three year period from 2003 to 2006.

However, static renderings of this sort suffer from the same problem that velocity vector images do; in regions with a high density of GNSS stations, tracks overlap significantly with one another, obscuring details. To solve this problem, our visualization lets the user interactively control the time range of interest, the amount of amplification and other settings. In addition, by animating the lines from start to finish, the user gets a real sense of motion that’s difficult to achieve in a static image.

We’ve applied our new visualization to the ~20 years of data from the GEONET array in Japan. Through it, we can see small but coherent changes in direction before and after the great 2011 Tohoku earthquake.

GPS data sets (in .json format) for both the GEONET data in Japan and the Plate Boundary Observatory (PBO) data in the western US are available at earthquake.rc.fas.harvard.edu.

This short animation shows many of the visualization’s interactive features. In order:

Modifying the multiplier adjusts how significantly the movements are magnified.
We can adjust the time slider nubs to select a particular time range of interest.
Using the map controls provided by the Google Maps JavaScript API, we can zoom into a tiny region of the map.
By enabling map markers, we can see information about individual GNSS stations.

By focusing on a stations of interest, we can even see curvature changes in the time periods before and after the event.

Station designated 960601 of Japan’s GEONET array is located on the island of Mikura-jima. Here we see the period from 2006 to 2012, with movement magnified 10^5.1 times (126,000x).

To achieve fast rendering of the line segments, we created a custom overlay using THREE.js to render the lines in WebGL. Data for the GNSS stations is passed to the GPU in a data texture, which allows our vertex shader to position each point on-screen dynamically based on user settings and animation.

We’re excited to continue this productive collaboration between Harvard and Google as we explore opportunities for groundbreaking, new earthquake visualizations. If you’d like to try out the visualization yourself, follow the instructions at earthquake.rc.fas.harvard.edu. It will walk you through the setup steps, including how to download the available data sets. If you’d like to report issues, great! Please submit them through the GitHub project page.

Acknowledgments

We wish to thank Bill Freeman, a researcher on Machine Perception, who hatched the idea and developed the initial prototypes, and Fernanda Viégas and Martin Wattenberg of the Big Picture Team for their visualization design guidance.

References

[1] Loveless, J. P., and Meade, B. J. (2010). Geodetic imaging of plate motions, slip rates, and partitioning of deformation in Japan, Journal of Geophysical Research.

By Jimbo Wilson, Software Engineer, Big Picture Team and Brendan Meade, Professor, Harvard Department of Earth and Planetary Sciences

Celebrating TensorFlow’s First Year

Open Source Programs Office — Wed, 09 Nov 2016 17:00:00 +0000

Originally posted on Google Research blog

It has been an eventful year since the Google Brain Team open-sourced TensorFlow to accelerate machine learning research and make technology work better for everyone. There has been an amazing amount of activity around the project: more than 480 people have contributed directly to TensorFlow, including Googlers, external researchers, independent programmers, students, and senior developers at other large companies. TensorFlow is now the most popular machine learning project on GitHub.

With more than 10,000 commits in just twelve months, we’ve made numerous performance improvements, added support for distributed training, brought TensorFlow to iOS and Raspberry Pi, and integrated TensorFlow with widely-used big data infrastructure. We’ve also made TensorFlow accessible from Go, Rust, and Haskell, released state-of-the-art image classification models – and answered thousands of questions on GitHub, StackOverflow, and the TensorFlow mailing list along the way.

At Google, TensorFlow supports everything from large-scale product features to exploratory research. We recently launched major improvements to Google Translate using TensorFlow (and Tensor Processing Units, which are special hardware accelerators for TensorFlow). Project Magenta is working on new reinforcement learning-based models that can produce melodies, and a visiting PhD student recently worked with the Google Brain team to build a TensorFlow model that can automatically interpolate between artistic styles. DeepMind has also decided to use TensorFlow to power all of their research – for example, they recently produced fascinating generative models of speech and music based on raw audio.

We’re especially excited to see how people all over the world are using TensorFlow. For example:

Australian marine biologists are using TensorFlow to find sea cows in tens of thousands of hi-res photos to better understand their populations, which are under threat of extinction.
An enterprising Japanese cucumber farmer trained a model with TensorFlow to sort cucumbers by size, shape, and other characteristics.
Radiologists have adapted TensorFlow to identify signs of Parkinson’s disease in medical scans.
Data scientists in the Bay Area have rigged up TensorFlow and the Raspberry Pi to keep track of the Caltrain.

We’re committed to making sure TensorFlow scales all the way from research to production and from the tiniest Raspberry Pi all the way up to server farms filled with GPUs or TPUs. But TensorFlow is more than a single open-source project – we’re doing our best to foster an open-source ecosystem of related software and machine learning models around it:

The TensorFlow Serving project simplifies the process of serving TensorFlow models in production.
TensorFlow “Wide and Deep” models combine the strengths of traditional linear models and modern deep neural networks.
For those who are interested in working with TensorFlow in the cloud, Google Cloud Platform recently launched Cloud Machine Learning, which offers TensorFlow as a managed service.

Furthermore, TensorFlow’s repository of models continues to grow with contributions from the community, with more than 3000 TensorFlow-related repositories are listed on GitHub alone! To participate in the TensorFlow community, you can follow our new Twitter account (@tensorflow), find us on GitHub, ask and answer questions on StackOverflow, and join the community discussion list.

Thanks very much to all of you who have already adopted TensorFlow in your cutting-edge products, your ambitious research, your fast-growing startups, and your school projects; special thanks to everyone who has contributed directly to the codebase. In collaboration with the global machine learning community, we look forward to making TensorFlow even better in the years to come!

By Zak Stone, Product Manager for TensorFlow

Google Summer of Code 2016 blog post round-up

Open Source Programs Office — Tue, 08 Nov 2016 15:00:00 +0000

We’re publishing guest posts from Google Summer of Code (GSoC) students, mentors and organizations every week and more are coming. Many have already written GSoC wrap-up posts on their own blogs, so we’ve rounded them up for you to explore.

“Static types in Python, oh my(py)!” by Tim Abbott, org admin for Zulip
“We posted mypy annotations as one of our project ideas for Google Summer of Code (GSoC). We found an incredible student, Eklavya Sharma, for the project. Eklavya did the vast majority of the hard work of annotating Zulip. Amazingly, he also found the time during the summer to migrate Zulip to use virtualenvs and then upgrade Zulip to Python 3!”

“A road from Google Summer of Code student to organization administrator” by Araz Abishov, org admin for HISP
“Google has created unprecedented opportunity both for young developers and open source communities, which I think everyone should take advantage of. GSoC is more than just a three months internship, and I hope that this post will be a good example of how it can change anyone’s life.”

“Summer of Code 2016: Wrapping it up” by Martin Braun, org admin for GNU Radio
“This summer was a great summer in terms of student participation. All three students will be presenting their work (either in person, or via poster) at this year’s GNU Radio Conference in Boulder, Colorado.”

“2016 Google Summer of Code Wrap-Up” by Ed Cable, org admin for Mifos Initiative
“Each year GSoC continues to unite and grow our community in different ways. Once again, we received incredibly valuable contributions to our Mifos X web and mobile clients this summer; most importantly we have cultivated numerous passionate contributors that will be a part of our community long into the future.”

“Road to GSoC 2016” by Minh Chu, student who worked on Neverland for KDE
“I was nervous about choosing a project. So many projects and requirements! After many hours, I finally decided to write a proposal for KDE’s Neverland Theme Builder and was accepted.”

“Git Rev News” by Christian Couder, mentor for Git
“Such performance improvements as well as the code consolidations around the sequencer are of course very nice. It is interesting and satisfying to see that they are the result of building on top of previous work over the years by GSoC students, mentors and reviewers.”

“Google Summer of Code 2016 Conclusion” by Amine Khaldi, org admin for ReactOS
“Students stumble upon many of the same difficulties ReactOS' own senior developers encountered during their early days, including that ever painful but necessary step to using a proper debugger instead of relying on printf statements in the code.”

“My Journey in Open Source / How to Get Started Contributing” by Nelson Liu, student who worked on scikit-learn for PSF
“The best way to get started is to simply jump in! There are a myriad of ways to contribute to an open source project. Obviously, writing code to fix bugs, add new features, or enhance existing ones are useful. However, you don't have to write code to help out!”

“Lasp and the Google Summer of Code” by Borja o’Cook, student who worked on Lasp for BEAM Community
“All in all, it's been an amazing experience. I've received a lot of support from my mentors and teammates; the Lasp team is full of incredible people.”

“GSoC 2016 Students in TEAMMATES” by Damith C. Rajapakse, org admin for TEAMMATES
“We had our biggest batch of students (7 students) in GSoC 2016, selected from 93 proposals, and representing 4 countries and 4 universities, working on TEAMMATES (an online feedback management system for education) and related sub projects.”

“User-friendly encryption now in Drupal 8!” by Colan Schwartz, mentor for Drupal
“There were several students interested in the topic, and wrote proposals to match. Talha Paracha's excellent proposal was accepted, and he began in earnest. With Adam Bergstein (nerdstein) and I mentoring him, Talha successfully worked through all phases of the project.”

“GSoC with Shogun” by Sanuj Sharma, student who worked on Shogun
“This was an excellent learning experience for me and I got to work with people from different countries (UK, Russia, Singapore, Germany) and cultures. I highly recommend students to participate in Google Summer of Code by looking for projects that interest them because having open source experience is highly beneficial, especially for programmers.”

We have wrap-up posts coming out every week so stay tuned for more. If you’re interested in participating in Google Summer of Code 2017, you can find details here.

By Josh Simmons, Open Source Programs Office

Announcing the Google Code-in 2016 mentor organizations

Open Source Programs Office — Mon, 07 Nov 2016 17:30:00 +0000

We’re excited to introduce the 17 open source organizations that are participating as mentor organizations for Google Code-in 2016. The contest, now in its seventh year, gives 13-17 year old pre-university students the opportunity to learn under the guidance of mentors by using their skills on real world applications, that is, open source projects.

Google Code-in officially starts for students on November 28, but students are encouraged to learn about the mentor organizations ahead of time and can get started by clicking on the links below.

Apertium - rule-based machine translation platform
BRL-CAD - computer graphics, 2D and 3D geometry modeling, and computer-aided design (CAD)
CCExtractor - open source tools for subtitle generation
Copyleft Games - building game development platforms for tomorrow
Drupal - content management platform
FOSSASIA - developing communities across all ages and borders to form a better future with Open Technologies and ICT
Haiku - operating system specifically targeting personal computing
KDE - team that creates Free Software for desktop and portable computing
MetaBrainz - builds community maintained databases
Mifos Initiative - transforming the delivery of financial services to the poor and the unbanked
MovingBlocks - like an open source Minecraft
OpenMRS - open source medical records system for the world
SCoRe - research lab that seeks sustainable solutions for problems faced by developing countries
Sugar Labs - learning platform and activities for elementary education
Systers - community for women involved in the technical aspects of computing
Wikimedia - non-profit foundation dedicated to bringing free content to the world, operating Wikipedia
Zulip - powerful, threaded open source group chat with apps for every major platform

Mentor organizations are currently creating thousands of tasks for students covering code, documentation, user interface, quality assurance, outreach, research and training. The contest officially starts for students on Monday, November 28th at 9:00am PST.

You can learn more about Google Code-in on the contest site where you’ll find Contest Rules, Frequently Asked Questions and Important Dates. There you’ll also find flyers and other helpful information including the Getting Started Guide. Our discussion mailing list is a great way to talk with other students, mentors and organization administrators about the contest. For questions about eligibility or other general questions, you can contact us at gci-support@google.com.

By Josh Simmons, Open Source Programs Office

Podcast to YouTube: an open source story

Open Source Programs Office — Fri, 04 Nov 2016 17:00:00 +0000

Almost a year ago Mark Mandel and I started the Google Cloud Platform Podcast, a weekly podcast that covers topics related to Google Cloud Platform, among other things. It's been a pretty successful podcast, but that’s not what I want to write about today.

After a while we started receiving emails from listeners that wanted to access our podcast on YouTube. Even though this might seem strange for those that love podcasts and have their favorite app on their phones, we decided that the customer is always right: we should post every episode to YouTube.

Specifications

Ok, so … how? Well, to create a video I need to merge the mp3 audio from an episode with a static image. Let's include the title of the episode and the Google Cloud Platform Podcast logo.

But once we post the video to YouTube we're going to need more than that! We need a description, some tags, and probably a link to the episode (SEO FTW!).

Where can we get that information from? Let's think about this for a minute. Where are others getting this information from? The RSS feed! Would it be possible to create a tool to which I could say "post the video for episode 46" and a couple minutes later the video appeared on YouTube? That'd be awesome! Let's do that!

Architecture

The application I wrote parses an RSS feed and given the episodes to publish it downloads the metadata and audio for an episode, generates the corresponding videos, and pushes them to YouTube.

Diagram of the flow of data in podcast-to-youtube

The hardest parts here are the creation of the image and the video. The rest is sending HTTP requests right and left.

Image Maker: rendering images in pure Go

After trying a couple of different tools I decided that the easiest was to create the image from scratch in Go using the image package from the standard library and a freetype library available on GitHub.

Probably the most fun part was to be able to choose a font that would make the title fit the image correctly regardless of the length in characters. I ended up creating a loop that:

chooses a font and measures the width of the resulting text
if it's too wide, decreases the font size by one and repeats.

Surprisingly, for me, this is actually a pretty common practice!

It is also worth mentioning the way I test the package: Using a standard image that I compare to the one generated by the package, then showing a "diff" image where all the pixels that differ are highlighted in red.

Diff image generated when using a wrong DPI.

The code for this package is available here.

Video maker: ffmpeg is awesome

From the beginning I knew I would end up using ffmpeg to create my video. Why? Well, because it is as simple as running this command:

$ ffmpeg -i image.png -i audio.mp3 video.mp4

Easy right? Well, this is once ffmpeg has been installed and correctly configured, which is actually not that simple and would make this tool hard to install on any machine.

That's why the whole tool runs on Docker. Docker is a pretty widespread technology, and thanks to Makefile I'm able to provide a tool that can be run like this:

$ make run

Conclusion

It took me a couple of days to write the tool and get it to a point where I could open source it, but it was totally worth it. I know that others will be able to easily reuse it, or even extend it. Who knows, maybe this should be exposed as a web application so anyone can use it, no Docker or Makefile needed!

I am currently using this tool weekly to upload the Google Cloud Platform Podcast episodes to this playlist, and you can find the whole code on this GitHub repository.

Any questions? I'm @francesc on Twitter.

By Francesc Campoy, Developer Advocate

Cilium: Networking and security for containers with BPF and XDP

Open Source Programs Office — Wed, 02 Nov 2016 17:00:00 +0000

This is a guest post by Daniel Borkmann who was recently recognized through the Google Open Source Peer Bonus program for his work on the Cilium project. We invited Daniel to share his project on our blog.

Our open source project, called Cilium, started as an experiment for Linux container networking tackling four requirements:

Scale: How can we scale in terms of addressing and with regards to network policy?
Extensibility: Can we be as extensible as user space networking in the Linux kernel itself?
Simplicity: What is an appropriate abstraction away from traditional networking?
Performance: Do we sacrifice performance in the process of implementing the aforementioned aspects?

We realize these goals in Cilium with the help of eBPF. eBPF is an efficient and generic in-kernel bytecode engine, that allows for full programmability. There are many subsystems in the Linux kernel that utilize eBPF, mainly in the areas of networking, tracing and security.

eBPF can be attached to key ingress and egress points of the kernel's networking data path for every network device. As input, eBPF operates on the kernel's network packet representation and can thus access and mangle various kinds of data, redirect the packet to other devices, perform encapsulations, etc.

This is a typical workflow: eBPF is programmed in a subset of C, compiled with LLVM which contains an eBPF back-end. LLVM then generates an ELF file containing program code, specification for maps and related relocation data. In eBPF, maps are efficient key/value stores in the kernel that can be shared between various eBPF programs, but also between user space. Given the ELF file, tools like tc (traffic control) can parse its content and load the program into the kernel. Before the program is executed, the kernel verifies the eBPF bytecode in order to make sure that it cannot affect the kernel's stability (e.g. crash the kernel and out of bounds access) and always terminates, which requires programs to be free of loops. Once it passed verification, the program is JIT (just-in-time) compiled.

Today, architectures such as x86_64, arm64, ppc64 and s390 have the ability to compile a native opcode image out of an eBPF program, so that instead of an execution through an in-kernel eBPF interpreter, the resulting image can run natively like any other kernel code. tc then installs the program into the kernel's networking data path, and with a capable NIC, the program can also be offloaded entirely into the hardware.

Cilium acts as a middle layer, plugs into container runtimes and orchestrators such as Kubernetes, Docker or CNI, and can generate and atomically update eBPF programs on the fly without requiring a container to restart. Thus, unlike connection proxies, an update of the datapath does not cause connections to be dropped. These programs are specifically tailored and optimized for each container, for example, a feature that a particular container does not need can just be compiled out and the majority of configuration becomes constant, allowing LLVM for further optimizations.

We have many implemented building blocks in Cilium using eBPF, such as NAT64, L3/L4 load balancing with direct server return, a connection tracker, port mapping, access control, NDisc and ARP responder and integration with various encapsulations like VXLAN, Geneve and GRE, just to name a few. Since all these building blocks run in the Linux kernel and have a stable API, there is of course no need to cross kernel/user space boundary, which makes eBPF a perfectly suited and flexible technology for container networking.

One step further in that direction is XDP, which was recently merged into the Linux kernel and allows for DPDK-like performance for the kernel itself. The basic idea is that XDP is tightly coupled with eBPF and hooks into a very early ingress path at the driver layer, where it operates with direct access to the packet's DMA buffer.

This is effectively as low-level as it can get to reach near-optimal performance, which mainly allows for tailoring high-performance load balancers or routers with commodity hardware. One advantage that comes with XDP is also that it reuses the kernel's security model for accessing the device as opposed to user space based mechanisms. It doesn't require any third party modules and works in concert with the Linux kernel. Both XDP and tc with eBPF are complementary to each other, and constitute a bigger piece of the puzzle for Cilium itself.

If you’re curious, check out the Cilium code or demos on GitHub.

By Daniel Borkmann, Cilium contributor

Using TensorFlow and JupyterHub in Classrooms

Open Source Programs Office — Mon, 31 Oct 2016 17:00:00 +0000

We’ve published a new solution and a companion GitHub repository that guides you through setting up a Google Container Engine cluster to run JupyterHub to automatically provision secure Jupyter containers for each user in a classroom or team. Don’t let the title of this article mislead you, not only does it use TensorFlow and JupyterHub, it’s actually an open source and cloud smorgasbord based on the Jupyter and Kubernetes platforms.

Jupyter is a powerful open source technology that gives you a platform to write and execute code to analyze, visualize and share the discoveries you find in your big data set. You can download a number of different Docker images preconfigured with many different notebook extensions and software packages to help you on any kind of data-science quest.

If you’re exploring on your own, and really want to get started quickly, you can get this all running on your local computer, but what if you want to take your expertise and lead a classroom of people along the same path? You have to either configure everything for them or walk them through configuring their own machines with all the required software.

This is where JupyterHub comes in, as a management layer in front of Jupyter instances, allowing you to configure users, using custom authentication, and giving you a Python interface to spawn new Jupyter instances for each user. Even with JupyterHub, you still need a way to provision physical and virtual hardware for the students.

Enter Kubernetes, an open source system for automating deploying, scaling and managing containerized applications. Google Container Engine is a fully managed service based on Kubernetes, allowing you to create clusters easily on Google Cloud Platform.

This solution comes with a JupyterHub Spawner class that allows it to create Kubernetes Pods, which are Docker images running Jupyter, for each user. It also comes with all the automation scripts required to create a Container Engine cluster and let you easily customize your setup.

When your students log into JupyterHub using Google OAuth2, they can choose from a list of several pre-built Jupyter images, including a newly updated “datalab-jupyter” image, which comes with the Google Datalab open source notebook extension enabling integration with BigQuery, Google Cloud ML, StackDriver, and it also has TensorFlow and the Apache Beam Python SDK for Google Cloud DataFlow installed. Users can also choose to run any of the pre-configured Jupyter docker-stack images, or you can build your own Docker images to run any special libraries or Jupyter configurations you want.

We hope that this solution allows you to get your classroom or team environment running quickly so you can focus on learning rather than configuring machines.

By Brad Svee, Cloud Solutions Architect

Dart in 2017 and beyond

Open Source Programs Office — Wed, 26 Oct 2016 09:00:00 +0000

We’re here at the Dart Developer Summit in Munich, Germany. Over 250 developers from more than 50 companies from all over the world just finished watching the keynote.

This is a summary of the topics we covered:

Dart is the fastest growing programming language at Google, with a 3.5x increase in lines of code since last year. We like to think that this is because of our focus on developer productivity: teams report 25% to 100% increase in speed of development. Google has bet its biggest business on Dart — the web apps built on Dart bring over $70B per year.

Google AdSense recently launched a ground-up redesign of their web app, built with Dart. Earlier this year, we announced that the next generation of AdWords is built with Dart. There are more exciting Dart products at Google that we’re looking forward to reveal. Outside Google, companies such as Wrike, Workiva, Soundtrap, Blossom, DG Logic, Sonar Design have all been using and enjoying Dart for years.

Our five year investment in this language is reaping fruit. But we’re not finished.

We learned that people who use Dart love its terse and readable syntax. So we’re keeping that.

We have also learned that Dart developers really enjoy the language’s powerful static analysis. So we’re making it better. With strong mode, Dart’s type system becomes sound (meaning that it rejects all incorrect programs). We’re also introducing support for generic methods.

We have validated that the programming language itself is just a part of the puzzle. Dart comes with ‘batteries included.’ Developers really like Dart’s core libraries — we will keep them tight, efficient and comprehensive. We will also continue to invest in tooling such as pub (our integrated packaging system), dartfmt (our automatic formatter) and, of course, the analyzer.

On the web, we have arrived at a framework that is an excellent fit for Dart: AngularDart. All the Google web apps mentioned above use it. It has been in production at Google since February. AngularDart is designed for Dart, and it’s getting better every week. In the past 4 months, AngularDart’s output has gotten 40% smaller, and our AngularDart web apps got 15% faster.

Today, we’re launching AngularDart 2.0 final. Tune in to the next session.

With that, we’re also releasing — as a developer preview — the AngularDart components that Google uses for its major web apps. These Material Design widgets are being developed by hundreds of Google engineers and are thoroughly tested. They are written purely in Dart.

We’re also making Dart easier to use with existing JavaScript libraries. For example, you will be able to use our tool to convert TypeScript .d.ts declarations into Dart libraries.

We’re making the development cycle much faster. Thanks to Dart Dev Compiler, compilation to JavaScript will take less than a second across all modern browsers.

We believe all this makes Dart an even better choice for web development than before. Dart has been here for a long time and it’s not going anywhere. It’s cohesive and dependable, which is what a lot of web developers want.

We’re also very excited about Flutter — a project to help developers build high-performance, high-fidelity, mobile apps for iOS and Android from a single codebase in Dart. More on that tomorrow.

We hope you’ll enjoy these coming two days. Tune in on the live stream or follow #dartsummit on Twitter.

By Filip Hracek, Developer Relations Program Manager

Google Summer of Code 2016 wrap-up: GNU Radio

Open Source Programs Office — Tue, 25 Oct 2016 17:00:00 +0000

This post is the third installment in our series of wrap-up posts reflecting on Google Summer of Code 2016. Check out the first and second posts in the series.

Originally posted on GNU Radio Blog

The summer has come to an end -- along with the Summer of Code for GNU Radio. It was a great season in terms of student participation, and as the students are preparing their last commits, this seems a good time to summarize their efforts.

All students presented their work (either in person, or via poster) at this year’s GNU Radio Conference in Boulder, Colorado.

gr-inspector

With gr-inspector, GNU Radio now has its own out-of-tree module, which serves as a repository for signal analysis algorithms, but also as a collection of fantastic examples. This module was created and worked on by Sebastian Müller, who was funded by Google Summer of Code (GSoC), and Christopher Richardson, who participated as a Summer of Code in Space (SOCIS) student funded by the European Space Agency. Sebastian also created a video demonstrating some of the features:

Both Sebastian and Chris have written up their efforts on their own blogs.

PyBOMBS GUI

Ravi Sharan was our other GSoC student, primarily working on a GUI for PyBOMBS, our installation helper tool. Ravi also worked on a bunch of other things, and has summarized his efforts as well.

The PyBOMBS GUI is written in Qt, and is a nice extension to our out-of-tree module ecosystem:

While some developers prefer the comfort of their command line environments, we hope that the PyBOMBS GUI will ease the entry for more new developers. The GUI ties in nicely with CGRAN, and with the correct setup, users can directly launch installation of out-of-tree modules from their browser.

Want to participate? Have ideas?

We will definitely apply for GSoC and SOCIS again next year! If you want to participate as a student, it helps a lot to get involved with the community early on. We also recommend you sign up for the mailing list, and get involved with GNU Radio by using it, reporting and fixing issues, or even publishing your own out-of-tree module. For more ideas, take a look at our summer of code wiki pages.

If you simply have ideas for future projects, those are welcome too! Suggest those on the mailing list, or simply edit the wiki page.

By Martin Braun, Organization Administrator for GNU Radio

Google Code-in 2016 now accepting organization applications

Open Source Programs Office — Mon, 24 Oct 2016 17:00:00 +0000

Google Code-in is our global online contest that invites pre-university students ages 13-17 to learn by contributing to open source software. The contest begins its 7th year on November 28th, 2016. With the start date of the contest rapidly approaching, we are now accepting applications for open source projects interested in being a part of Google Code-in.

Working with young students is a special responsibility and each year we hear inspiring stories from mentors who participate. To ensure these new, young contributors have a great support system, we select organizations that have gained experience in mentoring students by previously taking part in Google Summer of Code.

There were 14 organizations in 2015 that collectively created thousands of bite-sized tasks for students to choose from. Tasks are created in 5 categories:

Code: writing or refactoring
Documentation/Training: creating/editing documents and helping others learn more
Outreach/Research: community management, outreach/marketing, or studying problems and recommending solutions
Quality Assurance: testing and ensuring code is of high quality
User Interface: user experience research or user interface design and interaction

Once an organization is selected for Google Code-in 2016 they will define these tasks and recruit mentors who are interested in providing online support for students.

You can find a timeline, FAQ and other information about Google Code-in on our website. If you’re an educator interested in sharing Google Code-in with your students, you can find resources here.

By Josh Simmons, Open Source Programs Office

Budou: Automatic Japanese line breaking tool

Open Source Programs Office — Fri, 21 Oct 2016 17:00:00 +0000

Today we are pleased to introduce Budou, an automatic line breaking tool for Japanese. What is a line breaking tool and why is it necessary? English uses spacing and hyphenation as cues to allow for beautiful, aka more legible, line breaks. Japanese, which has none of these, is notoriously more difficult. Breaks occur randomly, usually in the middle of a word.

This is a long standing issue in Japanese typography on the web, and results in degradation of readability. We can specify the place which line breaks can occur with CSS coding, but this is a non-trivial manual process which requires Japanese vocabulary and knowledge of grammar.

Budou automatically translates Japanese sentences into organized HTML code with meaningful chunks wrapped in non-breaking markup so as to semantically control line breaks. Budou uses Cloud Natural Language API to analyze the input sentence, and it concatenates proper words in order to produce meaningful chunks utilizing PoS (part-of-speech) tagging and syntactic information. Budou outputs HTML code by wrapping the chunks in a SPAN tag. By specifying their display property as inline-block in CSS, semantic units will no longer be split at the end of a line.

Budou is a simple Python script that runs each sentence through the Cloud Natural Language API. It can easily be extended as a custom filter for template engines, or as a task for runners such as Grunt and Gulp. The latest version also caches the response so no duplicate requests are sent. If you are using Budou for a static website, you can process your HTML code before deployment.

Budou is aimed to be used in relatively short sentences such as titles and headings. Screen readers may read a sentence by splitting the chunks wrapped by SPAN tag or split by WBR tag, so it is discouraged to use Budou for body paragraphs.

As of October 2016, the Cloud Natural Language API supports English, Spanish, and Japanese, and Budou currently only supports Japanese. Support for other Asian languages with line break issues, such as Chinese and Thai, will be added as the API adds support.

Any comments and suggestions are welcome. You can find us on GitHub.

By Shuhei Iitsuka, UX Engineer

Introducing Nomulus: an open source top-level domain name registry

Open Source Programs Office — Tue, 18 Oct 2016 15:00:00 +0000

Today, Google is proud to announce the release of Nomulus, a new open source cloud-based registry platform that powers Google’s top level domains (TLDs). We’re excited to make this piece of Internet infrastructure available to everyone.

TLDs are the top level of the Internet Domain Name System (DNS), and they collectively host every domain name on the Internet. To manage a TLD, you need a domain name registry -- a behind-the-scenes system that stores registration details and DNS information for all domain names under that TLD. It handles WHOIS queries and requests to buy, check, transfer, and renew domain names. When you purchase a domain name on a TLD using a domain name registrar, such as Google Domains, the registrar is actually conducting business with that TLD’s registry on your behalf. That’s why you can transfer a domain from one registrar to another and have it remain active and 100% yours the entire time.

The project that became Nomulus began in 2011 when the Internet Corporation for Assigned Names and Numbers (ICANN) announced the biggest ever expansion of Internet namespace, aimed at improving choice and spurring innovation for Internet users. Google applied to operate a number of new generic TLDs, and built Nomulus to help run them.

We designed Nomulus to be a brand-new registry platform that takes advantage of the scalability and easy operation of Google Cloud Platform. Nomulus runs on Google App Engine and is backed by Google Cloud Datastore, a highly scalable NoSQL database. Nomulus can manage any number of TLDs in a single shared instance and supports the full range of TLD functionality required by ICANN, including the Extensible Provisioning Protocol (EPP), WHOIS, reporting, and trademark protection. It is written in Java and is released under the Apache 2.0 license.

We hope that by providing access to our implementation of core registry functions and up-and-coming services like Registration Data Access Protocol (RDAP), we can demonstrate advanced features of Google Cloud Platform and encourage interoperability and open standards in the domain name industry for registry operators like Donuts. With approximately 200 TLDs, Donuts has made early contributions to the Nomulus code base and has spun up an instance which they'll be sharing soon.

For more information, view Nomulus on GitHub.

By Ben McIlwain, Software Engineer

Google Summer of Code 2016 wrap-up: NRNB

Open Source Programs Office — Mon, 17 Oct 2016 17:00:00 +0000

This post is part of our series of Google Summer of Code wrap-ups, guest posts from students, mentors and organization admins reflecting on Google Summer of Code 2016. Don't miss our first post and follow along for more wrap-up posts and announcements.

We were so excited to be a part of Google Summer of Code (GSoC) again after a year off, we pulled together over 50 project ideas and dozens of eager mentors to develop open source code for network biology research. Organized as the National Resource for Network Biology (NRNB), we selected 15 proposals that brought together well-matched students, mentors and project ideas.

All 15 students passed their midterm and final evaluations, resulting in a wide range of (mostly) production-ready code, covering algorithm, UI, importer and converter development for both web and desktop for Cytoscape, cytoscape.js, SBML, SBGN, cBioPortal, Cell Designer, GraphSpace and more.

We are proud of the technical accomplishments and productivity of our students, and we are also proud of the many important aspects of diversity our students represent in the GSoC program, including geographical, gender and academic. Here are some numbers and facts about our 15 students compared to overall GSoC 2016 student stats in parentheses:

9 different countries, including 1 (of 2) from Croatia, 1 (of 3) from Armenia and 2 (of 12) from Turkey
20% female (compared to 12% overall)
67% Computer Science (compared to 78% overall), including PhD students in Biological Oceanography and Medical Biochemistry & Biotechnology, an MS student in Bioinformatics, and a pre-med undergraduate.

Here are some quotes and blogs from our students this year. If you are considering applying as student (or mentor) next year, here is some inspiration:

“I had the opportunity to learn and practice JavaScript with a very interesting project and having a mentor available was great for getting help when needed. The program seemed extremely well run and I would strongly recommend it to anyone interested.”

“Working in an NRNB [GSoC] training program helped to strengthen my resume and introduced me to the idea of combining a career in medicine with computer-based research.”

“I love the friendly atmosphere and the way the team works together. From the very beginning I [felt] well integrated in the group. It was pure fun to work together on the same project and to see how it [has] grown over the time. I [would] recommend everybody try the NRNB training program.”

Some of our student blogs:

Hovakim Grabski – "Java support for Deviser, a code generation system for SBML libraries"
Kaito Ii – "Interconvertible Layout software for CellDesigner"
Roman Schulte – "Offline SBML validation in the Java-based JSBML library"
Mridul Seth – "Import graphs in multiple formats and Cytoscape files into GraphSpace"

By Alex Pico and Kristina Hanspers, Organization Administrators for NRNB

Google Open Source Report Card

Open Source Programs Office — Fri, 14 Oct 2016 17:00:00 +0000

Open source software enables Google to build things quickly and efficiently without reinventing the wheel, allowing us to focus on solving new problems. We stand on the shoulders of giants and we know it. This is why we support open source and make it easy for Googlers to release the projects they’re working on internally as open source.

Today we’re sharing our first Open Source Report Card, highlighting our most popular projects, sharing a few statistics and detailing some of the projects we’ve released in 2016.

We’ve open sourced over 20 million lines of code to date and you can find a listing of some of our best known project releases on our website. Here are some of our most popular projects:

Android - a software stack for mobile devices that includes an operating system, middleware and key applications.
Chromium - a project encompassing Chromium, the software behind Google Chrome, and Chromium OS, the software behind Google Chrome OSdc devices.
Angular - a web application framework for JavaScript and Dart focused on developer productivity, speed and testability.
TensorFlow - a library for numerical computation using data flow graphics with support for scalable machine learning across platforms from data centers to embedded devices.
Go - a statically typed and compiled programming language that is expressive, concise, clean and efficient.
Kubernetes - a system for automating deployment, operations and scaling of containerized applications.
Polymer - a lightweight library built on top of Web Components APIs for building encapsulated re-usable elements in web applications.
Protobuf - an extensible, language-neutral and platform-neutral mechanism for serializing structured data.
Guava - a set of Java core libraries that includes new collection types (such as multimap and multiset), immutable collections, a graph library, functional types, an in-memory cache, and APIs/utilities for concurrency, I/O, hashing, primitives, reflection, string processing and much more.
Yeoman - a robust and opinionated set of scaffolding tools including libraries and a workflow that can help developers quickly build beautiful and compelling web applications.

While it’s difficult to measure the full scope of open source at Google, we can use the subset of projects that are on GitHub to gather some interesting data. Today our GitHub footprint includes over 84 organizations and 3,499 repositories, 773 of which were created this year.

Googlers use countless languages from Assembly to XSLT, but what are their favorites? GitHub flags the most heavily used language in a repository and we can use that to find out. A survey of GitHub repositories shows us these are some of the languages Googlers use most often:

JavaScript
Java
C/C++
Go
Python
TypeScript
Dart
PHP
Objective-C
C#

Many things can be gleaned using the open source GitHub dataset on BigQuery, like usage of tabs versus spaces and the most popular Go packages. What about how many times Googlers have committed to open source projects on GitHub? We can search for Google.com email addresses to get a baseline number of Googler commits. Here’s our query:

SELECT count(*) as n
FROM [bigquery-public-data:github_repos.commits]
WHERE committer.date > '2016-01-01 00:00'
AND REGEXP_EXTRACT(author.email, r'.*@(.*)') = 'google.com'

With this, we learn that Googlers have made 142,527 commits to open source projects on GitHub since the start of the year. This dataset goes back to 2011 and we can tweak this query to find out that Googlers have made 719,012 commits since then. Again, this is just a baseline number as it doesn’t count commits made with other email addresses.

Looking back at the projects we’ve open-sourced in 2016 there’s a lot to be excited about. We have released open source software, hardware and datasets. Let’s take a look at some of this year’s releases.

Seesaw
Seesaw is a Linux Virtual Server (LVS) based load balancing platform developed in Go by our Site Reliability Engineers. Seesaw, like many projects, was built to scratch our own itch.

From our blog post announcing its release: “We needed the ability to handle traffic for unicast and anycast VIPs, perform load balancing with NAT and DSR (also known as DR), and perform adequate health checks against the backends. Above all we wanted a platform that allowed for ease of management, including automated deployment of configuration changes.”

Vendor Security Assessment Questionnaire (VSAQ)
We assess the security of hundreds of vendors every year and have developed a process to automate much of the initial information gathering with VSAQ. Many vendors found our questionnaires intuitive and flexible, so we decided to share them. The VSAQ Framework includes four extensible questionnaire templates covering web applications, privacy programs, infrastructure as well as physical and data center security. You can learn more about it in our announcement blog post.

OpenThread
OpenThread, released by Nest, is a complete implementation of the Thread protocol for connected devices in the home. This is especially important because of the fragmentation we’re seeing in this space. Development of OpenThread is supported by ARM, Microsoft, Qualcomm, Texas Instruments and other major vendors.

Magenta
Can we use machine learning to create compelling art and music? That’s the question that animates Magenta, a project from the Google Brain team based on TensorFlow. The aim is to advance the state of the art in machine intelligence for music and art generation and build a collaborative community of artists, coders and machine learning researchers. Read the release announcement for more information.

Omnitone
Virtual reality (VR) isn’t nearly as immersive without spatial audio and much of VR development is taking place on proprietary platforms. Omnitone is an open library built by members of the Chrome Team that brings spatial audio to the browser. Omnitone builds on standard Web Audio APIs to deliver an immersive experience and can be used alongside projects like WebVR. Find out more in our blog post announcing the project’s release.

Science Journal
Today’s smartphones are packed with sensors that can tell us interesting things about the world around us. We launched Science Journal to help educators, students and citizen scientists tap into those sensors. You can learn more about the project in our announcement blog post.

Cartographer
Cartographer is a library for real-time simultaneous localization and mapping (SLAM) in 2D and 3D with Robot Operating System (ROS) support. Combining data from a variety of sensors, this library computes positioning and maps surroundings. This is a key element of self-driving cars, UAVs and robotics as well as efforts to map the insides of famous buildings. More information on Cartographer can be found in our blog post announcing its release.

This is just a small sampling of what we’ve released this year. Follow the Google Open Source Blog to stay apprised of Google’s open source software, hardware and data releases.

By Josh Simmons, Open Source Programs Office

Google Summer of Code 2016 wrap-up: HUES Platform

Open Source Programs Office — Wed, 12 Oct 2016 15:47:00 +0000

Every year Google Summer of Code pairs university students with mentors to hone their skills while working on open source projects, and every year we like to post wrap-ups from the open source projects about their experience and what students accomplished. Stay tuned for more!

The Holistic Urban Energy Simulation (HUES) platform is an open source platform for facilitating the design and control of renewables-based distributed energy systems. The platform is an initiative of the Urban Energy Systems Laboratory at Empa in Switzerland, in collaboration with our research partners at ETH-Zurich, EPFL, the University of Geneva and the Lucerne University of Applied Sciences. As we push towards the second version of the HUES platform, we had help from three bright and enthusiastic students as part of the Google Summer of Code (GSoC).

Project 1: Real-time wind flow in cities

Air flow pattern around a building configuration (left); link to Rhinoceros/Grasshopper (middle & right)

People in cities are suffering more and more from scorching heat, caused by global warming and bad urban planning. This traps heat inside cities and has led to soaring air conditioning demand, making cities even hotter - a vicious circle! Clever bioclimatic urban design can mitigate urban heat by facilitating the use of natural ventilation and guiding air streams. However, the simulation of wind flow is a computationally and technically demanding task. There is a need to provide urban planners and architects with a tool able to predict wind flow patterns in real-time to facilitate development of energy efficient and passive designs.

Lukas Bystricky, a student at Florida State University, developed a Fast Fluid Dynamics (FFD) library in C# exactly for this purpose. Lukas’s implementation is based on the paper by Jos Stam (1999). In contrast to the original implementation, where a cell centred finite difference is used to discretize the equations, Lukas applies a staggered grid finite difference, which is the standard finite difference in Computational Fluid Dynamics (CFD). This is done to prevent spurious pressure oscillations near the boundary which can occur in cell centered finite difference for the Navier-Stokes equations. This does not change much in the algorithm or solvers, but makes enforcing the boundary conditions significantly more complicated. So far, Lukas uses a simple Jacobi solver as linear solver, as was the case in Stam's original implementation, but he plans to replace it with more efficient solvers in the future. Also, he is validating his library with typical benchmarks.

We are now coupling Lukas’s library into our HUES platform, more specifically into the 3D CAD software Rhinoceros and its visual programming platform Grasshopper. The final goal is to have an intuitive real-time visual design tool of wind flow for urban planners and architects. Also, we will use it to couple it to whole year dynamic building energy simulation programs, to better capture microclimatic effects of the urban context in simulating building energy consumption of designs.

Project 2: Modular energy hub modeling framework

A connection between two bus objects in a CopyHub container

Distributed energy system components are modular in nature and interact across multiple scales. As such, there is a need for a modeling framework that can easily construct and configure systems of modular entities (energy demands, sources, converters, storages and network links) across scales. Frederik Banis, a student at the University of Applied Sciences Stuttgart, developed a modular approach to modeling distributed multi-energy systems (energy hubs) in Python, based on the Open Energy System Modelling Framework (Oemof) and Pyomo.

In the developed framework, energy systems components are specified in a common format allowing for easy duplication and reconfiguring at larger scales. The platform enables easy manipulation of an energy hub grouping multiple components (demand, sources: electricity, natural gas; systems: photovoltaic panels, wind turbines, gas boils, combined heat and power engines, etc.), as well as copying it (from hub1 to hub2) to create a larger interlinked system (district) where multiple energy hubs are connected. This hierarchical nested structure can be repeated as needed, and detailed results about the energy supply of each technology or energy stream can be analyzed in the form of different plots for each system or sub-system.

Project 3: Open source energy simulation database

The HUES platform includes a growing array of datasets describing the technical and economic characteristics of distributed energy technologies. Currently, this data is stored in separate modules using different data structures and file formats, making it difficult to explore holistically and query systematically. To address this, GSoC student Khushboo Mandlecha has developed an open source database to enable the linked exploration, querying and visualization of data in the platform.

The first part of the project involved the development of server based scripts to automatically extract relevant data from the modules of the existing HUES platform, and write this data to a common database. A standard format for technology component data was developed, enabling users to upload technology data files to be stored in the new database. The new database has been developed in MongoDB, enabling fast data retrieval and allowing everything to be retrieved in the form of JSON objects. The second part of the project involved the development of a web-based portal for querying, visualizing and downloading data. Once this portal is complete, it will be possible to visualize the contents of the database in different ways, enabling users to get a sense of the distribution of property values and facilitating the identification of outliers. Ultimately, the database will help researchers and practitioners using the HUES platform to develop models and perform comprehensive analyses of distributed energy systems.

By L. Andrew Bollinger, Julien Marquant and Christoph Waibel; Urban Energy Systems Laboratory, Empa, Switzerland

Announcing Google Code-in 2016 and Google Summer of Code 2017

Open Source Programs Office — Mon, 10 Oct 2016 17:00:00 +0000

One of the goals of the Open Source Programs Office is to encourage more people to contribute to open source software. One way we achieve that goal is through our student programs, Google Summer of Code (for university students) and Google Code-in (for pre-university students).

Over 15,000 students from more than 100 countries have worked with 23,000 mentors and contributed to 560+ open source projects.

This is why we’re excited to announce the next round of both of our student programs!

Google Code-in begins for students November 28, 2016

For the seventh consecutive year, Google Code-in will give students (ages 13-17) a chance to explore open source. Students will find opportunities to learn and get hands on experience with tasks from a range of categories. This structure allows students to stretch themselves as they take on increasingly more challenging tasks.

Getting started is easy: once the contest begins, simply choose an interesting task from our participating organizations’ lists and complete it. Mentors from the organizations are available to help online.

Google Code-in is for students asking questions like:

What is open source?
What kinds of stuff do open source projects do?
How can I write real code when all I’ve done is a little classroom work?
Can I contribute even if I’m not really a programmer?

With tasks in five different categories, there’s something to fit almost any student’s skills:

Code: writing or refactoring
Documentation/Training: creating/editing documents and helping others learn more
Outreach/research: community management, outreach/marketing, or studying problems and recommending solutions
Quality Assurance: testing and ensuring code is of high quality
User Interface: user experience research or user interface design and interaction

Google Summer of Code student applications open on March 20, 2017

Google Summer of Code (GSoC) provides university students from around the world with an opportunity to take their skills and hone them by contributing to open source projects during their summer break from university.

Students gain invaluable experience working with mentors on these open source software projects, earning a stipend upon successful completion of their project.

We’re proud to keep this tradition going: we’ll be opening student applications for Google Summer of Code 2017 on March 20, 2017. Applications for interested open source organizations open on January 19, 2017.

Students, it’s never too early to start preparing or thinking about your proposal. You can learn about the organizations that participated in Google Summer of Code 2016 and the projects students worked on. We also encourage you to explore other resources like the student and mentor manuals and frequently asked questions.

You can learn more on the program website.

Share the news with your friends and stay tuned, more details are coming soon!

By Josh Simmons, Open Source Programs Office

An open source font system for everyone

Open Source Programs Office — Thu, 06 Oct 2016 17:00:00 +0000

Originally posted on the Google Developers Blog

A big challenge in sharing digital information around the world is “tofu”—the blank boxes that appear when a computer or website isn’t able to display text: ⯐. Tofu can create confusion, a breakdown in communication, and a poor user experience.

Five years ago we set out to address this problem via the Noto—aka “No more tofu”—font project. Today, Google’s open source Noto font family provides a beautiful and consistent digital type for every symbol in the Unicode standard, covering more than 800 languages and 110,000 characters.

A few samples of the 110,000+ characters covered by Noto fonts.

The Noto project started as a necessity for Google’s Android and Chrome OS operating systems. When we began, we did not realize the enormity of the challenge. It required design and technical testing in hundreds of languages, and expertise from specialists in specific scripts. In Arabic, for example, each character has four glyphs (i.e., shapes a character can take) that change depending on the text that comes after it. In Indic languages, glyphs may be reordered or even split into two depending on the surrounding text.

The key to achieving this milestone has been partnering with experts in the field of type and font design, including Monotype, Adobe, and an amazing network of volunteer reviewers. Beyond “no more tofu” in the common languages used every day, Noto will be used to preserve the history and culture of rare languages through digitization. As new characters are introduced into the Unicode standard, Google will add these into the Noto font family.

Google has a deep commitment to openness and the accessibility and innovation that come with it. The full Noto font family, design source files, and the font building pipeline are available for free at the links below. In the spirit of sharing and communication across borders and cultures, please use and enjoy!

Noto fonts download: https://www.google.com/get/noto/
Design source files: https://github.com/googlei18n/noto-source
Font building pipeline: https://github.com/googlei18n/fontmake

By Xiangye Xiao and Bob Jung, Internationalization

Introducing Cartographer

Open Source Programs Office — Wed, 05 Oct 2016 17:00:00 +0000

We are happy to announce the open source release of Cartographer, a real-time simultaneous localization and mapping (SLAM) library in 2D and 3D with ROS support.

SLAM algorithms combine data from various sensors (e.g. LIDAR, IMU and cameras) to simultaneously compute the position of the sensor and a map of the sensor’s surroundings. For example, consider this approach to drawing a floor plan of your living room:

Grab a laser rangefinder, stand in the middle of the room, and draw an X on a piece of paper.
Measure the distance from where you’re standing to any wall.
Draw a line on the paper where the wall is and write down the distance between the X (your position) and the wall.
Measure the distance from where you’re standing to another wall and add it to the drawing as well.
Now, move to another part of the room.
Since the walls (hopefully) haven’t moved, you can measure your distance to the same two walls to determine your new position.

SLAM is an essential component of autonomous platforms such as self driving cars, automated forklifts in warehouses, robotic vacuum cleaners, and UAVs.

Cartographer builds globally consistent maps in real-time across a broad range of sensor configurations common in academia and industry. The following video is a demonstration of Cartographer’s real-time loop closure:

A detailed description of Cartographer’s 2D algorithms can be found in our ICRA 2016 paper.

Thanks to ROS integration and support from external contributors, Cartographer is ready to use on several robot platforms with ROS support:

At Google, Cartographer has enabled a range of applications from mapping museums and transit hubs to enabling new visualizations of famous buildings.

We recognize the value of high quality datasets to the research community. That’s why, thanks to cooperation with the Deutsches Museum (the largest tech museum in the world), we are also releasing three years of LIDAR and IMU data collected using our 2D and 3D mapping backpack platforms during the development and testing of Cartographer.

Our focus is on advancing and democratizing SLAM as a technology. Currently, Cartographer is heavily focused on LIDAR SLAM. Through continued development and community contributions, we hope to add both support for more sensors and platforms as well as new features, such as lifelong mapping and localizing in a pre-existing map.

By Damon Kohler, Wolfgang Hess, and Holger Rapp, Google Engineering

Introducing the Open Images Dataset

Open Source Programs Office — Mon, 03 Oct 2016 16:00:00 +0000

Originally posted on the Google Research Blog

In the last few years, advances in machine learning have enabled Computer Vision to progress rapidly, allowing for systems that can automatically caption images to apps that can create natural language replies in response to shared photos. Much of this progress can be attributed to publicly available image datasets, such as ImageNet and COCO for supervised learning, and YFCC100M for unsupervised learning.

Today, we introduce Open Images, a dataset consisting of ~9 million URLs to images that have been annotated with labels spanning over 6000 categories. We tried to make the dataset as practical as possible: the labels cover more real-life entities than the 1000 ImageNet classes, there are enough images to train a deep neural network from scratch and the images are listed as having a Creative Commons Attribution license^*.

The image-level annotations have been populated automatically with a vision model similar to Google Cloud Vision API. For the validation set, we had human raters verify these automated labels to find and remove false positives. On average, each image has about 8 labels assigned. Here are some examples:

Annotated images form the Open Images dataset. Left: Ghost Arches by Kevin Krejci. Right: Some Silverware by J B. Both images used under CC BY 2.0 license

We have trained an Inception v3 model based on Open Images annotations alone, and the model is good enough to be used for fine-tuning applications as well as for other things, like DeepDream or artistic style transfer which require a well developed hierarchy of filters. We hope to improve the quality of the annotations in Open Images the coming months, and therefore the quality of models which can be trained.

The dataset is a product of a collaboration between Google, CMU and Cornell universities, and there are a number of research papers built on top of the Open Images dataset in the works. It is our hope that datasets like Open Images and the recently released YouTube-8M will be useful tools for the machine learning community.

By Ivan Krasin and Tom Duerig, Software Engineers

* While we tried to identify images that are licensed under a Creative Commons Attribution license, we make no representations or warranties regarding the license status of each image and you should verify the license for each image yourself.

Another year of Haskell Hacking in the Google Zurich Office

Open Source Programs Office — Fri, 30 Sep 2016 16:00:00 +0000

For the fourth time, the Google Open Source Programs Office have co-sponsored a three-day hackathon for Haskell, an open source functional programming language. Gleb Peregud from Google’s Zurich office talks more about the event below.

On the weekend of July 22nd, 126 Haskell enthusiasts got together for another installment of ZuriHac, a yearly open source Haskell hackathon held in Zurich, Switzerland, and like the last two years it took place at Google Zurich.

Participants could either hack uninterrupted in the main room or listen to a number of presentations in the tech talk room. Each day was kicked off with a keynote — Bas van Dijk talked about the use of functional programming at LumiGuide (slides), Edward Kmett about monad homomorphisms, and Andres Löh about generic-sop, a new approach to generic programming. All three talks drew a full room of interested listeners.

We also prepared two codelabs for Haskell beginners, and rallied 28 dedicated volunteers to serve as mentors (thank you all!) so that there was always someone to ask for help.

Aside from keynotes, there were five other talks: an experience report on parallelizing and distributing scientific computations (slides), an overview of a language to program FPGAs called CλaSH, an interactive tour through low-level pieces of the GHC Haskell compiler, an introduction to web programming using Spock (slides) and a talk on revamping the build system of GHC (slides).

Spontaneous mini-lectures attract smaller crowds.

As is traditional, after a full Saturday of hacking, we went out to barbecue down by the Zurich lake.

We were lucky that weather forecast was mistaken about a looming thunderstorm!

We managed to beat last year's record, and welcomed 126 attendees. We hope to have even more participants next year - our goal is to host 150 hackers in 2017. See you in a year!

By Gleb Peregud, Site Reliability Engineer

.NET and PowerShell tooling for the Google Cloud Platform

Open Source Programs Office — Thu, 29 Sep 2016 17:15:00 +0000

Last month Google made an announcement unveiling support for Visual Studio, C#, PowerShell, Microsoft SQL Server and more on the Google Cloud Platform. With so many new features, it is easy to gloss over some of the technical aspects of the announcement, especially the fact that all of the developer tooling and libraries are open source and available on GitHub.

This post will go into some of the details behind the new C# libraries, PowerShell cmdlets, and Visual Studio extension. All three products are open source, have an exciting roadmap for the future and are hungry for your feedback.

C# bindings for Google APIs

Source: https://github.com/googlecloudplatform/google-cloud-dotnet
Docs: https://cloud.google.com/dotnet/

For years, Google has had innovative technologies powering its data centers, unfortunately Google’s internal APIs and technology couldn’t directly benefit you and your software. That was, until the Google Cloud Platform started exposing public APIs for things like machine learning, storage, logging etc. With these APIs publicly available, you can add powerful capabilities to your apps without needing to manage complex infrastructure.

There have been C# bindings for Google APIs for years. In fact, Google receives hundreds of millions of API calls from C# clients every day. But newer APIs, especially those from the Google Cloud Platform, require more advanced features like bidirectional streaming. That’s why rather than using HTTP/REST many newer Google APIs are built on top of gRPC, a high performance, open source universal RPC framework.

But don’t worry, we have C# bindings for those gRPC-based APIs too; all of it open source and on GitHub.

In both cases, the client library is the result of a C# code generator. We take the API’s discovery document (analogous to a WSDL) and generate C# code. gRPC APIs require more careful design than other APIs, but the end product is the same. Once built, the API libraries are published to NuGet.

C# code generators for Google APIs isn’t the entire story.

Source code generated from tools can look foreign at times. So for libraries where the codegen isn’t good enough, we have hand-written wrappers to provide a better, more idiomatic experience. In some cases -- such as CRUD operations using the Datastore API -- the hand-written library cuts down on the required lines of code by half.

Finally, support for C# doesn’t just mean code. We are also working to ensure Google APIs are supported on different runtimes too. Most Google APIs work on the cross-platform .NET Core runtime and we are continuing to expand support.

PowerShell support

Source: https://github.com/googlecloudplatform/google-cloud-powershell
Docs: http://googlecloudplatform.github.io/google-cloud-powershell/

C# support is great when you are writing full applications, but for DevOps, scripting is more typical. The Cloud SDK provides command-line tools (gcloud, gsutil) for managing cloud resources, but when running on Windows, Windows PowerShell is a dramatically more productive environment. Google Cloud tools for PowerShell is a set of cmdlets so you can manage your Google cloud resources. They are strongly typed, and integrate seamlessly with other PowerShell tools. For example, to learn more about a cmdlet, just use Get-Help.

In designing the PowerShell cmdlets, the main goal was to be idiomatic. We wanted to follow the best practices and guidelines so PowerShell novices and pros alike could use our cmdlets. Of course, if we have anything wrong, please log an issue on the GitHub repository. Pull requests are also welcome.

Visual Studio

Source: https://github.com/googlecloudplatform/google-cloud-visualstudio
Docs: https://cloud.google.com/visual-studio/

The C# and PowerShell features should help developers using Google services. But the biggest impact on developer productivity comes from being inside the Visual Studio IDE.

From within Visual Studio you can search for new extensions and find the Google Cloud Platform Extension for Visual Studio. It provides tools for viewing/managing data stored in Google Cloud Storage and Google Cloud SQL. It also provides support for deploying ASP.NET 4.x applications to Google Compute Engine.

It is only the first release and we have some big plans for the future. You can see a lot of the short-term features we have planned by looking at the issues list in GitHub. Like making Google APIs light up for the new .NET Core runtime, being able to deploy ASP.NET Core applications to Google App Engine or Google Container Engine will be huge. Stay tuned for a future blog post about how to run C# on Google App Engine Flexible Environment, as well.

We’re just getting started

Hopefully you share my enthusiasm for Google’s ongoing development in .NET tooling. Not only is it exciting to be able to take advantage of Google Cloud Platform technologies, but also to see a future where .NET Core enables C# code to run cross-platform.

But to be successful we need your help.

If you have questions, be sure to ask on Stack Overflow (e.g. the google-cloud-visualstudio or google-cloud-powershell tags). If you have problems, please open issues on GitHub (libraries, VS, PowerShell). If you still have trouble, participate in the google-cloud-dev group.

The team here at Google is thrilled to be working with the .NET stack and your feedback is immensely helpful in prioritizing things.

By Chris Smith, Software Engineer

A sizzling open source release for the Australian Election site

Open Source Programs Office — Wed, 28 Sep 2016 22:49:00 +0000

Originally posted on the Geo Developers Blog

One of the best parts of my job at Google is 20 percent time. While I was hired to help developers use Google’s APIs, I value the time I'm afforded to be a student myself—to learn new technologies and solve real-world problems. A few weeks prior to the recent Australian election an opportunity presented itself. A small team in Sydney set their sights on helping the 15 million voters stay informed of how to participate, track real-time results, and (of course) find the closest election sausage sizzle!

Our team of designers, engineers and product managers didn't have an immediate sense of how to attack the problem. What we did have was the power of Google’s APIs, programming languages, and Cloud hosting with Firebase and Google Cloud Platform.

The result is a mish-mash of some technologies we'd been wanting to learn more about. We're open sourcing the ausvotes.withgoogle.com repository to give developers a sense of what happens when you get a handful of engineers in a room with a clear goal and a immovable deadline.

The Election AU 2016 repository uses:

Go from Google App Engine instances to serve the appropriate level of detail for users' viewport queries from memory at very low latency, and
Dart to render the live result maps on top of Google Maps JavaScript API using Firebase real time database updates.

A product is only as good as the attention and usage is receives. Our team was really happy with the results of our work:

406,000 people used our maps, including 217,000 on election day.
We had 139 stories in the media.
Our map was also embedded in major news websites, such as Sky News.

Complete setup and installation instructions are available in the GitHub README.

By Brett Morgan, Developer Programs Engineer

Google Open Source Peer Bonus Program

Open Source Programs Office — Fri, 23 Sep 2016 17:00:00 +0000

Five years ago the Open Source Programs Office established the Open Source Peer Bonus Program to remind Googlers of the importance of the myriad developers outside of Google who keep open source healthy and growing.

The program works like this: we invite Googlers to nominate open source developers outside of the company who deserve recognition for their contributions to interesting open source projects including those used by Google. After review by a team of volunteer engineers, the recipients receive our heartfelt thanks and a small token of our appreciation.

We have recognized more than 500 open source developers from 30+ countries who have contributed their time and talent to over 400 open source projects.

Having just finished the latest round of the program, we’d like to recognize the individuals and the projects they worked on. Here’s everyone who gave us permission to thank them publicly:

Name	Project	Name	Project
Olli Etuaho	ANGLE	Alexander Morozov	Go programming language
Minko Gechev	Angular	Joel Sing	LibreSSL
Georgios Kalpakas	Angular	Daniel Borkmann	Linux kernel
Spencer Low	AOSP (Android)	Michael Ellerman	Linux kernel
Holden Karau	Apache Spark	Heiko Stuebner	Linux kernel
Dave Taht	Bufferbloat	Jonathan Garbee	Material Design Lite
Leon Han	Chromium	Chris Sullo	Nikto
Yoav Weiss	Chromium	Carl Friedrich Bolz	PyPy
Rob Wu	Chromium	Brett Cannon	Python
Faisal Vali	Clang	Raymond Hettinger	Python
Matt Godbolt	Compiler Explorer	Tim Peters	Python
Paul Kocialkowski	coreboot	Tully Foote	ROS
Jonathan Kollasch	coreboot	Igor Babuschkin	TensorFlow
Nicolas Reinecke	coreboot	Yuan Tang	TensorFlow
Werner Zeh	coreboot	Hanno Boeck	The Fuzzing Project
Daniel Greenfeld	Django	Khaled Hosny	TruFont
Eric Whitney	ext4	Tom Rini	U-Boot
Ben Martin	FontForge	Caitlin Potter	V8
Dmitri Shuralyov	go-github	Brian Behlendorf	ZFS on Linux

Congratulations all and thank you so much for your contributions to the open source community!

By Helen Hu, Open Source Programs Office

Angular, version 2: proprioception-reinforcement

Open Source Programs Office — Thu, 22 Sep 2016 23:02:00 +0000

Originally posted on the Angular Blog

Today, at a special meetup at Google HQ, we announced the final release version of Angular 2, the full-platform successor to Angular 1.

What does "final" mean? Stability that's been validated across a wide range of use cases, and a framework that's been optimized for developer productivity, small payload size, and performance. With ahead-of-time compilation and built-in lazy-loading, we’ve made sure that you can deploy the fastest, smallest applications across the browser, desktop, and mobile environments. This release also represents huge improvements to developer productivity with the Angular CLI and styleguide.

Angular 1 first solved the problem of how to develop for an emerging web. Six years later, the challenges faced by today’s application developers, and the sophistication of the devices that applications must support, have both changed immensely. With this release, and its more capable versions of the Router, Forms, and other core APIs, today you can build amazing apps for any platform. If you prefer your own approach, Angular is also modular and flexible, so you can use your favorite third-party library or write your own.

From the beginning, we built Angular in collaboration with the open source development community. We are grateful to the large number of contributors who dedicated time to submitting pull requests, issues, and repro cases, who discussed and debated design decisions, and validated (and pushed back on) our RCs. We wish we could have brought every one of you in person to our meetup so you could celebrate this milestone with us tonight!

What’s next?

Angular is now ready for the world, and we’re excited for you to join the thousands of developers already building with Angular 2. But what’s coming next for Angular?

A few of the things you can expect in the near future from the Angular team:

Bug fixes and non-breaking features for APIs marked as stable
More guides and live examples specific to your use cases
More work on animations
Angular Material 2
Moving WebWorkers out of experimental
More features and more languages for Angular Universal
Even more speed and payload size improvements

Semantic Versioning

We heard loud and clear that our RC labeling was confusing. To make it easy to manage dependencies on stable Angular releases, starting today with Angular 2.0.0, we will move to semantic versioning. Angular versioning will then follow the MAJOR.MINOR.PATCH scheme as described by semver:

the MAJOR version gets incremented when incompatible API changes are made to stable APIs,
the MINOR version gets incremented when backwards-compatible functionality are added,
the PATCH version gets incremented when backwards-compatible bug are fixed.

Moving Angular to semantic versioning ensures rapid access to the newest features for our component and tooling ecosystem, while preserving a consistent and reliable development environment for production applications that depend on stability between major releases, but still benefit from bug fixes and new APIs.

Contributors

Aaron Frost, Aaron (Ron) Tsui, Adam Bradley, Adil Mourahi, agpreynolds, Ajay Ambre, Alberto Santini, Alec Wiseman, Alejandro Caravaca Puchades, Alex Castillo, Alex Eagle, Alex Rickabaugh, Alex Wolfe, Alexander Bachmann, Alfonso Presa, Ali Johnson, Aliaksei Palkanau, Almero Steyn, Alyssa Nicoll, Alxandr, André Gil, Andreas Argelius, Andreas Wissel, Andrei Alecu, Andrei Tserakhau, Andrew, Andrii Nechytailov, Ansel Rosenberg, Anthony Zotti, Anton Moiseev, Artur Meyster, asukaleido, Aysegul Yonet, Aziz Abbas, Basarat Ali Syed, BeastCode, Ben Nadel, Bertrand Laporte, Blake La Pierre, Bo Guo, Bob Nystrom, Borys Semerenko, Bradley Heinz, Brandon Roberts, Brendan Wyse, Brian Clark, Brian Ford, Brian Hsu, dozingcat, Brian Yarger, Bryce Johnson, CJ Avilla, cjc343, Caitlin Potter, Cédric Exbrayat, Chirayu Krishnappa, Christian Weyer, Christoph Burgdorf, Christoph Guttandin, Christoph Hoeller, Christoffer Noring, Chuck Jazdzewski, Cindy, Ciro Nunes, Codebacca, Cody Lundquist, Cody-Nicholson, Cole R Lawrence, Constantin Gavrilete, Cory Bateman, Craig Doremus, crisbeto, Cuel, Cyril Balit, Cyrille Tuzi, Damien Cassan, Dan Grove, Dan Wahlin, Daniel Leib, Daniel Rasmuson, dapperAuteur, Daria Jung, David East, David Fuka, David Reher, David-Emmanuel Divernois, Davy Engone, Deborah Kurata, Derek Van Dyke, DevVersion, Dima Kuzmich, Dimitrios Loukadakis, Dmitriy Shekhovtsov, Dmitry Patsura, Dmitry Zamula, Dmytro Kulyk, Donald Spencer, Douglas Duteil, dozingcat, Drew Moore, Dylan Johnson, Edd Hannay, Edouard Coissy, eggers, elimach, Elliott Davis, Eric Jimenez, Eric Lee Carraway, Eric Martinez, Eric Mendes Dantas, Eric Tsang, Essam Al Joubori, Evan Martin, Fabian Raetz, Fahimnur Alam, Fatima Remtullah, Federico Caselli, Felipe Batista, Felix Itzenplitz, Felix Yan, Filip Bruun, Filipe Silva, Flavio Corpa, Florian Knop, Foxandxss, Gabe Johnson, Gabe Scholz, GabrielBico, Gautam krishna.R, Georgii Dolzhykov, Georgios Kalpakas, Gerd Jungbluth, Gerard Sans, Gion Kunz, Gonzalo Ruiz de Villa, Grégory Bataille, Günter Zöchbauer, Hank Duan, Hannah Howard, Hans Larsen, Harry Terkelsen, Harry Wolff, Henrique Limas, Henry Wong, Hiroto Fukui, Hongbo Miao, Huston Hedinger, Ian Riley, Idir Ouhab Meskine, Igor Minar, Ioannis Pinakoulakis, The Ionic Team, Isaac Park, Istvan Novak, Itay Radotzki, Ivan Gabriele, Ivey Padgett, Ivo Gabe de Wolff, J. Andrew Brassington, Jack Franklin, Jacob Eggers, Jacob MacDonald, Jacob Richman, Jake Garelick, James Blacklock, James Ward, Jason Choi, Jason Kurian, Jason Teplitz, Javier Ros, Jay Kan, Jay Phelps, Jay Traband, Jeff Cross, Jeff Whelpley, Jennifer Bland, jennyraj, Jeremy Attali, Jeremy Elbourn, Jeremy Wilken, Jerome Velociter, Jesper Rønn-Jensen, Jesse Palmer, Jesús Rodríguez, Jesús Rodríguez, Jimmy Gong, Joe Eames, Joel Brewer, John Arstingstall, John Jelinek IV, John Lindquist, John Papa, John-David Dalton, Jonathan Miles, Joost de Vries, Jorge Cruz, Josef Meier, Josh Brown, Josh Gerdes, Josh Kurz, Josh Olson, Josh Thomas, Joseph Perrott, Joshua Otis, Josu Guiterrez, Julian Motz, Julie Ralph, Jules Kremer, Justin DuJardin, Kai Ruhnau, Kapunahele Wong, Kara Erickson, Kathy Walrath, Keerti Parthasarathy, Kenneth Hahn, Kevin Huang, Kevin Kirsche, Kevin Merckx, Kevin Moore, Kevin Western, Konstantin Shcheglov, Kurt Hong, Levente Morva, laiso, Lina Lu, LongYinan, Lucas Mirelmann, Luka Pejovic, Lukas Ruebbelke, Marc Fisher, Marc Laval, Marcel Good, Marcy Sutton, Marcus Krahl, Marek Buko, Mark Ethan Trostler, Martin Gontovnikas, Martin Probst, Martin Staffa, Matan Lurey, Mathias Raacke, Matias Niemelä, Matt Follett, Matt Greenland, Matt Wheatley, Matteo Suppo, Matthew Hill, Matthew Schranz, Matthew Windwer, Max Sills, Maxim Salnikov, Melinda Sarnicki Bernardo, Michael Giambalvo, Michael Goderbauer, Michael Mrowetz, Michael-Rainabba Richardson, Michał Gołębiowski, Mikael Morlund, Mike Ryan, Minko Gechev, Miško Hevery, Mohamed Hegazy, Nan Schweiger, Naomi Black, Nathan Walker, The NativeScript Team, Nicholas Hydock, Nick Mann, Nick Raphael, Nick Van Dyck, Ning Xia, Olivier Chafik, Olivier Combe, Oto Dočkal, Pablo Villoslada Puigcerber, Pascal Precht, Patrice Chalin, Patrick Stapleton, Paul Gschwendtner, Pawel Kozlowski, Pengfei Yang, Pete Bacon Darwin, Pete Boere, Pete Mertz, Philip Harrison, Phillip Alexander, Phong Huynh, Polvista, Pouja, Pouria Alimirzaei, Prakal, Prayag Verma, Rado Kirov, Raul Jimenez, Razvan Moraru, Rene Weber, Rex Ye, Richard Harrington, Richard Kho, Richard Sentino, Rob Eisenberg, Rob Richardson, Rob Wormald, Robert Ferentz, Robert Messerle, Roberto Simonetti, Rodolfo Yabut, Sam Herrmann, Sam Julien, Sam Lin, Sam Rawlins, Sammy Jelin, Sander Elias, Scott Hatcher, Scott Hyndman, Scott Little, ScottSWu, Sebastian Hillig, Sebastian Müller, Sebastián Duque, Sekib Omazic, Shahar Talmi, Shai Reznik, Sharon DiOrio, Shannon Ayres, Shefali Sinha, Shlomi Assaf, Shuhei Kagawa, Sigmund Cherem, Simon Hürlimann (CyT), Simon Ramsay, Stacy Gay, Stephen Adams, Stephen Fluin, Steve Mao, Steve Schmitt, Suguru Inatomi, Tamas Csaba, Ted Sander, Tero Parviainen, Thierry Chatel, Thierry Templier, Thomas Burleson, Thomas Henley, Tim Blasi, Tim Ruffles, Timur Meyster, Tobias Bosch, Tony Childs, Tom Ingebretsen, Tom Schoener, Tommy Odom, Torgeir Helgevold, Travis Kaufman, Trotyl Yu, Tycho Grouwstra, The Typescript Team, Uli Köhler, Uri Shaked, Utsav Shah, Valter Júnior, Vamsi V, Vamsi Varikuti, Vanga Sasidhar, Veikko Karsikko, Victor Berchet, Victor Mejia, Victor Savkin, Vinci Rufus, Vijay Menon, Vikram Subramanian, Vivek Ghaisas, Vladislav Zarakovsky, Vojta Jina, Ward Bell, Wassim Chegham, Wenqian Guo, Wesley Cho, Will Ngo, William Johnson, William Welling, Wilson Mendes Neto, Wojciech Kwiatek, Yang Lin, Yegor Jbanov, Zach Bjornson, Zhicheng Wang, and many more...

With gratitude and appreciation, and anticipation to see what you'll build next, welcome to the next stage of Angular.

By Jules Kremer, Angular Team

Introducing OpenType Font Variations

Open Source Programs Office — Wed, 14 Sep 2016 09:15:00 +0000

Cześć and hello from the ATypI conference in Warsaw! Together with Microsoft, Apple and Adobe, we’re happy to announce the launch of variable fonts as part of OpenType 1.8, the newest version of the font standard. With variable fonts, your device can display text in myriads of weights, widths, or other stylistic variations from a single font file with less space and bandwidth.

OpenType variable fonts support OpenType Layout variation.

To prevent that the $ sign becomes a black blob,

the stroke disappears at a certain weight.

At Google, we started tinkering with variable fonts about two years ago. We were fascinated by the typographic opportunities, and we got really excited when we realized that variable fonts would also help to save space and bandwidth. We proposed reviving Apple’s TrueType GX variations in OpenType, and started experimenting with it in our tools. The folks at Microsoft then started a four-way collaboration between Microsoft, Apple, Adobe, and Google, together with experts from type foundries and tool makers. Microsoft did the spec work; Apple brought their existing technology and expertise; Adobe updated their CFF format into CFF2; and we brought the tools and testing we’d been developing. After months of intense polishing, the specification is now finished.

On the Google end, we did a lot of work to build, edit and display variable fonts:

implemented most of the spec in FontTools
updated the fontmake pipeline so variable fonts can be built from common source formats
updated HarfBuzz
worked with Adobe to implement CFF2 in FreeType
fixed bugs in FreeType
developed a demo tool FontView

As always, all our font tools are free and open source for everyone to use and contribute.

Now that the spec is public, we can finish the work by merging the changes upstream so that our code will soon flow into products. We’ll also update Noto to support variations (for many writing systems, the sources are already there — the rest will follow). Much more work lies ahead, for example, implementing variations in Google Fonts. Together with other browser makers, we’re already working on a proposal to extend CSS fonts with variations. Once everyone agrees on the format, we’ll support it in Google Chrome. And there are many other challenges ahead, like incorporating font variations into other Google products—so it will be a busy time for us! We are incredibly excited that an amazing technology from 23 years ago is coming back to life again today. Huge thanks to our friends at Adobe, Apple, and Microsoft for a great collaboration!

To learn more, read Introducing OpenType Variable Fonts, or talk to us at the FontTools group.

By Behdad Esfahbod and Sascha Brawer, Fonts and Text Rendering, Google Internationalization

Google Summer of Code 2016 statistics: celebrating our mentors

Open Source Programs Office — Fri, 09 Sep 2016 17:00:00 +0000

Our final statistics post of the year is dedicated to to the incredible Google Summer of Code (GSoC) 2016 mentors. There were a total of 2,524 mentors, but today we'll look at the 1,500+ mentors who were assigned to an active project. Mentors are the lifeblood of our program. Without their hard work and dedication to the success of our students, there would be no GSoC. A merry band of volunteers, mentors work with students for more than 12 weeks — remotely, across multiple time zones, giving their time, expertise and guidance in addition to a regular full-time job for an average of 7.45 hours a week. Today we’ll take a closer look at our 2016 team.

GSoC 2016 mentors reside all over the world and represent 66 countries.

Want to see the data? Here’s the breakdown of the countries our mentors come from.

We have many mentors who participate in GSoC year after year. In 2016, we have six mentors who have participated since the program’s inception in 2005! GSoC “lifer” Bart Massey, who participated as a mentor for Portland State University and X.Org had this to say about his time with GSoC:

“I'm not sure which is more astonishing, that I am 12 years older with GSoC or that GSoC is 12 years old with me. Some of the most fantastic, interesting, brilliant and hardworking folks on the planet have gotten together every year for 12 years to change the world: Google folks and open source leadership and skilled, special students. It's been great to get to be part of it all, both as Portland State University and during my time with X.Org...I hope I get to keep working with and hanging out with these people I love every year forever.”

Awww, we love you too Bart!

There are also plenty of newbies to the program each year and 2016 is no exception. We’d like to welcome 528 (33%) new mentors to the GSoC family.

Some fun facts:

Average age: 32
Youngest: 14
Oldest: 78
Most common mentor first name: David

At the end of each program year, we invite two mentors from each participating organization to join us at the Mentor Summit, a three day unconference at Google HQ in Northern California. There they enjoy a weekend with their peers to talk about all things open source-y (a technical term) and have some fun.

A huge thanks to each and every Google Summer of Code mentor. We salute you.

By Mary Radomile, Open Source Programs

Google Summer of Code 2016 is a wrap

Open Source Programs Office — Fri, 02 Sep 2016 17:00:00 +0000

As school in much of the world begins, Google Summer of Code 2016 winds down. The last three months have seen a whirlwind of activity on over 178 open source projects. University students from around the globe have been working with their mentors to contribute their technical skills to the common good.

Student participants submitted their completed work late last month, mentors evaluated the submissions, and the results have been announced.

We accepted 1,206 university students from 67 countries in April and we are excited to announce that 1,032 students (85.6%) successfully completed the program. To learn more about how that compares to previous years, check out our statistics from the last eleven years.

Google Summer of Code isn’t over though. In October we’ll be hosting our annual mentor summit in Sunnyvale, California where mentors and organization administrations will meet and exchange ideas.

Thank you to all of the students, mentors and organization administrators for your contributions to open source and for making the 12th year of Google Summer of Code such a great success!

By Josh Simmons, Open Source Programs Office

From Summer of Code to Game of Thrones on the back of a JavaScript Dragon (Part 3)

Open Source Programs Office — Fri, 26 Aug 2016 17:45:00 +0000

This guest post is a part of a short series about Tatyana Goldberg, Guy Yachdav and Christian Dallago and the journey that was inspired by their participation as Google Summer of Code mentors for the BioJS project. Check out the first and second posts in the series.

This blog post marks the end of our short series following our adventures in open source. As you may recall, it all started thanks to Google Summer of Code (GSoC) which brought our team together. The GSoC collaboration spurred us to start a class at Technical University of Munich (TUM) that eventually took on the Game of Thrones data science project which became an international sensation.

The success of our Game of Thrones project opened a lot of doors. First, we were invited to participate in the Morpheus Cup which is a prestigious university olympiad that brings together students from all over Europe to compete in digital challenges.

Our team rocked the competition winning two challenges and making it to the finalist stage in the third challenge. We were honored to represent our university and grateful for Google’s sponsorship of our team.

The students and mentors of the Game of Thrones project at the Morpheus Cup challenge in May 2016. From left to right: Georgi Anastasov, Emiliyana Kalinova, Maximilian Bandle (all students), Guy Yachdav (mentor), Christian Dallago (mentor), Tobias Piffrader, Theodor Chesleran (both students) and Tatyana Goldberg (mentor).

Another opportunity that followed was an invitation to speak at a TEDx event at TUM on July 28th, 2016. In the event, titled “The Common Extraordinary,” Guy presented our work with data mining as bioinformaticians, sharing how we’ve made the field of data science accessible to our students and how we helped popularize it through the Game of Thrones project.

More speaking engagements are already scheduled: at meetups, coffee talks and conferences where we plan to keep evangelizing data mining and tell the story of our open source adventure.

What’s next? We’re excited to continue as mentors and org admins in GSoC and to carry on teaching data science and JavaScript at the university. In between classes and our daily research work we’re now being asked by friends, family members, colleagues and even strangers whether we can help them use data mining to answer questions on subjects ranging from politics, science, sports and even their personal lives.

Just the other day we were approached with the idea of developing an app that would take in a set of personality traits, process them along with social network data and help in suggesting life decisions: Should I date that person? Should I really take this job? Is Baltimore the city for me?

That interest goes even beyond our personal circles. A recent trade media report pointed out that by using machine learning in an unexpected context, the Game of Thrones project demonstrated the disruptive force of data mining. This force, the article continues, could make an impact on the next industrial revolution - Industry 4.0 - where data plays a key role.

Do you have interesting questions you’d like to answer or a data set you’d like to make predictions with? Curious about BioJS or our JavaScript course? Please reach out to us on Twitter or in the comments.

In the near future we dream of starting our own consultancy, as we already have requests from companies that want our help with upcoming data science projects. It seems our team has found its entrepreneurial bent!

We hope enjoyed this trilogy of blog posts, that our story has inspired you and that you too will continue to adventure in open source and collaborative development. If you’re not involved with Google Summer of Code, consider joining. It’s a great way to build up your project and share it with the world. More importantly, it lets you work with amazing people with whom, as we learned, it is possible to reach the sky.

By Tatyana Goldberg, Christian Dallago, and Guy Yachdav, BioJS

Stories from Google Code-in: OpenMRS and SCoRe

Open Source Programs Office — Mon, 22 Aug 2016 17:00:00 +0000

Google Code-in is our annual contest that gives students age 13 to 17 experience in computer science through contributions to open source projects. This blog post is the third installment in our series reflecting on the experiences of Google Code-in 2015 grand prize winners. Be sure to check out the first and second posts in the series, too.

In this post we look at the stories of three more Google Code-in (GCI) grand prize winners. Our grand prize winners come from a pool of 980 students from 65 countries who, all told, completed 4,776 tasks for 14 open source projects.

We were lucky enough to host many of these extraordinary young coders at Google HQ for a few days this summer. Over that time, we learned more about where they came from, what they gained by participating in GCI and what they plan to do as new members of the open source community.

Google Code-in 2015 Grand Prize Winners explore the SF Bay Area in this immersive Google Street View display with fellow open source program managers Stephanie Taylor and Cat Allman who run GCI.

Our first story today is that of Břetislav Hájek from the Czech Republic, who chose to work with the OpenMRS project because he sees their work as important. OpenMRS is an open source medical record system that improves healthcare delivery in resource-constrained regions.

Břetislav got into computer science through web development, so he started by working on tasks related to HTML and CSS. This gave him confidence to take on more challenging tasks. His favorite task was creating a web application for searching through patients. While he didn’t find it hard, he learned a lot and was proud to have made something useful. Reflecting on Google Code-in, Břetislav said: “That's the thing I like about GCI. I always treat tasks as opportunities to learn something new. And the learning is more entertaining since I work on real problems.”

IRC communication proved to be an important part of Břetislav’s success. Other students were there and tried to help each other out as best they could, and there were always mentors available to help guide them. He enjoyed the friendly environment. The community motivated him to work harder and try new things. In the end, Břetislav was glad to have participated and is motivated to continue his work.

Next we have Vicente Bermudez from Uruguay who discovered Google Code-in through a story in the local news celebrating a Uruguayan grand prize winner from a previous year. Like Břetislav, Vicente chose to work on the OpenMRS project because the cause spoke to him.

He got into programming through his love of video games and his desire to create his own. He hadn’t heard about programming before but initial research piqued his interest. Following his curiosity, he learned Java and expanded his knowledge from there. Conveniently, much of OpenMRS is built with Java!

The task-based structure worked well for Vicente. He was unsure of some tasks, recognizing that he didn’t know much about what they required. For instance, he hesitated to take on one that involved creating a Windows Phone app because he had never created a mobile app. But he persisted and, five days later, he had completed it and learned a lot about mobile development.

It surprised Vicente how much he learned in such a short time span. He had this to say: “During the contest I gained knowledge in a variety of fields such as programming, testing, video editing, and graphic design. The mentors encouraged us to think about quality instead of quantity, and I learned a lot from that.”

Vicente loved his Google Code-in experience and plans to continue contributing to open source projects, especially OpenMRS.

The last student story we’ll share today is that of Anesu Mafuvadze, a student from the US who worked with the Sustainable Computing Research Group (SCoRe). His introduction to computer science came through robotics in one of his high school classes which used a language similar to C++.

Anesu was thrilled by the experience of bringing the robots to life with code. He described his introduction this way: “The more I programmed the more captivated I became; I loved how easily I could convert my wildest ideas into fully functioning programs; I loved the thrill of working in an environment that demands minute precision; above all, I loved creating programs that other people found useful.”

Online documentation and YouTube tutorials fueled Anesu’s education for several years as he picked up multiple languages and began participating in programming contests. But he knew something was missing, Anesu lacked real world coding experience and had never collaborated with others. As such, he didn’t pay much attention to the readability of his code, wasn’t aware of version control, didn’t write extensive tests and had never built something for the common good.

Enter Google Code-in. Working with mentors helped Anesu deliver quality and building open source software required him to learn collaboration tools and value readability. The contest also gave him an opportunity to build on skills that he hadn’t developed, such as web development. Anesu says the experience made him a better programmer and that the introduction to open source has motivated him to use his skills on projects that benefit society.

Thank you to Břetislav, Vicente and Anesu for their hard work contributing to open source projects and for sharing their stories with us. We have one more blog post coming with more student stories so stay tuned!

By Josh Simmons, Open Source Programs Office

Opening up Science Journal

Open Source Programs Office — Fri, 19 Aug 2016 16:56:00 +0000

Science Journal is an app that turns your Android phone into a mobile science tool, allowing you to use the sensors in your phone to explore the world around you. The Making & Science team launched Science Journal a few months ago at Bay Area Maker Faire 2016 and have been excited to see different projects people have done with it all over the world!

Today we are happy to announce that we are releasing Science Journal 1.1 on the Google Play Store and also publishing the core source for the app. Open source software and hardware has been hugely beneficial to the science education ecosystem. By open sourcing, we’ll be able to improve the app faster and also to provide the community with an example of a modern Android app built with Material Design principles.

One important feature in Science Journal is the ability to connect to external devices over Bluetooth LE. We have open source firmware which runs on several Arduino microcontrollers already. In the near future, we will provide alternate ways to get your sensor data into Science Journal: stay tuned (or follow along with our commits)!

We believe that anyone can be a scientist anywhere. Science doesn’t just happen in the classroom or lab. Tools like Science Journal let you see how the world works with just your phone and now you can explore how Science Journal itself works, too. Give it a try and let us know what you think!

By Justin Koh, Software Engineer

A Google Santa Tracker update from Santa’s Elves

Open Source Programs Office — Wed, 17 Aug 2016 17:00:00 +0000

Originally posted on the Google Developers Blog

By Sam Thorogood, Developer Programs Engineer

Today, we're announcing that the open source version of Google's Santa Tracker has been updated with the Android and web experiences that ran in December 2015. We extended, enhanced and upgraded our code, and you can see how we used our developer products - including Firebase and Polymer - to build a fun, educational and engaging experience.

To get started, you can check out the code on GitHub at google/santa-tracker-weband google/santa-tracker-android. Both repositories include instructions so you can build your own version.
Santa Tracker isn’t just about watching Santa’s progress as he delivers presents on December 24. Visitors can also have fun with the winter-inspired experiences, games and educational content by exploring Santa's Village while Santa prepares for his big journey throughout the holidays.
Below is a summary of what we’ve released as open source.

Android app

The Santa Tracker Android app is a single APK, supporting all devices, such as phones, tablets and TVs, running Ice Cream Sandwich (4.0) and up. The source code for the app can be found here.
Santa Tracker leverages Firebase features, including Remote Config API, App Invites to invite your friends to play along, and Firebase Analytics to help our elves better understand users of the app.
Santa’s Village is a launcher for videos, games and the tracker that responds well to multiple devices such as phones and tablets. There's even an alternative launcher based on the Leanback user interface for Android TVs.

Games on Santa Tracker Android are built using many technologies such as JBox2D (gumball game), Android view hierarchy (memory match game) and OpenGL with special rendering engine (jetpack game). We've also included a holiday-themed variation of Pie Noon, a fun game that works on Android TV, your phone, and inside Google Cardboard's VR.

Android Wear

The custom watch faces on Android Wear provide a personalized touch. Having Santa or one of his friendly elves tell the time brings a smile to all. Building custom watch faces is a lot of fun but providing a performant, battery friendly watch face requires certain considerations. The watch face source code can be found here.
Santa Tracker uses notifications to let users know when Santa has started his journey. The notifications are further enhanced to provide a great experience on wearables using custom backgrounds and actions that deep link into the app.

On the web

Santa Tracker is mobile-first: this year's experience was built for the mobile web, including an amazing brand new, interactive - yet fully responsive, village: with three breakpoints, touch gesture support and support for the Web App Manifest.
To help us develop Santa at scale, we've upgraded to Polymer 1.0+. Santa Tracker's use of Polymer demonstrates how easy it is to package code into reusable components. Every housein Santa's Village is a custom element, only loaded when needed, minimizing the startup cost of Santa Tracker.

Many of the amazing new games (like Present Bounce) were built with the latest JavaScript standards (ES6) and are compiled to support older browsers via the Google Closure Compiler.
Santa Tracker's interactive and fun experience is enhanced using the Web Animations API, a standardized JavaScript APIfor unifying animated content.
We simplified the Chromecast support this year, focusing on a great screensaver that would countdown to the big event on December 24th - and occasionally autoplay some of the great video content from around Santa's Village.

We hope that this update inspires you to make your own magical experiences based on all the interesting and exciting components that came together to make Santa Tracker!

From Google Summer of Code to Game of Thrones on the Back of a JavaScript Dragon (Part 2)

Open Source Programs Office — Fri, 12 Aug 2016 22:16:00 +0000

This guest post is a part of a short series about Guy Yachdav, Tatyana Goldberg and Christian Dallago and the journey that was inspired by their participation as Google Summer of Code mentors for the BioJS project. Don’t miss the first post in the series. Heads up, this post contains spoilers for Game of Thrones seasons 5 and 6!

We built on the Google Summer of Code (GSoC) philosophy and the lessons we learned from participating in 2014 by starting a JavaScript Technology class at the Technical University of Munich (TUM).

We began with two dozen students who worked on expanding the BioJS visualization library. Our class became popular quickly and the number of applicants doubled each semester (nearly 180 applicants for 40 seats in the 2016 summer term).

In 2016 our team grew to include Christian Dallago, who had joined as a GSoC mentor. Together we decided to break with tradition of our course’s previous semesters. Instead of focusing on data visualization, we wanted to introduce students to data science with JavaScript. To get our students fully engaged, we decided the project would center on data from the hit TV show, Game of Thrones.

Our aim was to create an online portal for Game of Thrones fans which would:

Provide the most comprehensive, structured and open data set about the Game of Thrones world accessible via API.
Present an interactive map based on JavaScript.
Listen to what people are saying on Twitter about each of the show’s characters.
Use machine learning algorithms to predict the likelihood of each character’s death.

Our plan worked — the students were engaged. It was a beautiful sight to see: GitHub repos humming with activity as each dev team delved deeper into their projects. As a project manager, you know you’ve got something good when issues are being opened and closed at 4:00 AM!

The results were mind blowing. In 50 days of programming, 36 students opened over 1,200 issues and pull requests, pushed 3,300 commits, released four apps to NPM, and, of course, produced one absolutely amazing website.

The website amasses data from 2,028 characters. Our map shows 240 landmarks and the paths traveled by 28 characters. Our Twitter sentiment analysis tool analyzed over 3 million tweets. And we launched the first ever machine learning-based prediction algorithm that predicts the likelihood of dying for the 1,451 characters in the show that are still alive.

Visualization of Twitter sentiment analysis data for Jon Snow during season 5 of Game of Thrones. The X axis shows the timeline and the Y axis shows the number of positive (green) and negative (red) tweets. Each tweet is analyzed by an algorithm using a neural network to determine whether the tweet’s writer has a positive, negative or neutral attitude toward the character.

Since launch, the site’s popularity has skyrocketed. Following our press release, we were covered by over 1,500 media outlets, most notably Time, The Guardian, Rolling Stone, Daily Mail, BBC, Reuters, The Telegraph, CNET and many more. HowStuffWorks, The Vulture and others produced videos about the site and Chris Hardwick’s Comedy Central show did a segment about us. We've also given countless interviews to TV, radio and newspapers.


	Google Analytics for the website. Left chart shows the number of visitors to the website during the first week after launch, reaching over 73K visitors on April 25th. Right chart shows the number of visitors at a given time point during the same week.

The most exciting part of the project was predicting the likelihood that any given character would die using machine learning. Machine learning algorithms find rules and patterns in the data, things that humans cannot obviously and simply detect. Once the rules and patterns are identified, we apply machine learning to make inferences or predictions from novel, previously unseen, data sets.

Warning: The next paragraphs contain spoilers for seasons 5 and 6 of Game of Thrones!

In order to predict the likelihood of a character’s death, we collected information about all of the characters that appeared in books 1 to 5 and analyzed over 30 features, including age, gender, marital status and others. Then we used a support vector machine (SVM) to statistically compare the features of characters, both dead and alive, to predict who would get the axe next. Our prediction was correct for 74% of all cases and surprised us by placing a number of characters thought to be relatively safe in grave danger.

According to our predictions, Jon Snow, who was seemingly betrayed and murdered by fellow members of the Night’s Watch at the end of season 5, had only an 11% chance of dying. Indeed, Jon has risen from the dead in the second episode of season 6! We also predicted that the rulers of Dorn (Doran and Trystane) Martell are at a high likelihood of death and, as predicted, they were taken out in the first episode of the new season.

Of course, as is always the case with predictions, there were also misses. We didn’t expect Roose Bolton to be killed off nor did we see Hodor’s departure coming.

This experience was an amazing ride for our team and it all started with Google Summer of Code! In the next post we’ll share what followed and where we see ourselves heading in the future.

By Guy Yachdav, Tatyana Goldberg and Christian Dallago, BioJS

Which languages convey the most information in the least space? Introducing the Unimorph dataset.

Open Source Programs Office — Mon, 08 Aug 2016 17:00:00 +0000

Several years ago a science journalist asked me which languages could pack the most information into a 140-character Tweet. Because Twitter defines a character roughly as a single Unicode code point, this turns out to be an easy question to answer. Chinese almost certainly rates as the most “compact” language from that point of view because a single Chinese character represents a whole morpheme (in linguist terminology, a minimal unit of meaning) whereas an English letter only represents a part of a morpheme. The Chinese equivalent of I don’t eat meat, which in English takes 16 characters including spaces is 我不吃肉, which takes just four.

But this question relates to a broader question that as a linguist I have often been asked: which languages are the most “efficient” at conveying information? Or, which languages can convey the same information in the smallest amount of space? Untethered by the idiosyncrasies of Twitter, this question becomes quite difficult to answer. What do you mean by “space”? Number of characters? Number of bytes? Number of syllables? Each of these has its own problems. And perhaps more crucially, what do you mean by “information”? The Shannon notion of information does not straightforwardly apply here.

A group of us at Google set out to answer this question, or at least to provide the form that an answer would have to take. We had the resources and experience needed to annotate data in multiple languages, and we were able to divert some of those resources to this task. The results were published in a paper presented at the 2014 International Conference on Language Resources and Evaluation in Reykjavík, Iceland.

We are now releasing the data on GitHub. The data consist of 85 sentences typical of the kinds of sentences generated by Google Now, translated into eight typologically diverse languages: English, French, Italian, German, Russian, Arabic, Korean, Chinese, which include some highly inflected and uninflected languages, and various types of morphology including inflectional and agglutinative. The data were annotated by one to three annotators depending on the language, with morphological information, counts of the marked features and other information. The main data file is in HTML, color coded by language, which makes it easy to browse but also easy to extract into other formats.

Since the basic information conveyed by each sentence can be assumed to be the same across languages, the main focus of the research was on the additional information that each language marks, and cannot avoid marking. For example, the English sentence:

Use my location for the search results and other services.

has the French translation:

Utilisez ma position pour les résultats de recherche et d'autres services.

The verb ending -ez, in boldface above marks “addressee respect”, a bit of information that is missing from the English original. One could have used a different ending on the French verb, but then that would not avoid this bit of information—it would be choosing to mark lack of respect, or familiarity with the addressee.

In the paper we tried various ways of measuring the differing information content of the languages relative to various definitions of “space”. Considering all the factors together, we concluded that the languages that conveyed the most information in a given amount of space were highly inflected languages like Russian, with uninflected languages like Chinese actually being the “least efficient” at conveying information.

We don’t expect this to be the final answer, which is why we are releasing the data as open source in the hopes that others will find it useful and maybe can even extend it to more sentences or a wider variety of languages. Ultimately though, any answer to the question of which languages convey the most information in the smallest amount of space must seriously address what is meant by “information”, and must pay heed to the famous maxim by the Russian linguist Roman Jakobson (1959) that “languages differ essentially in what they must convey and not in what they may convey.”

By Richard Sproat, Research Scientist

Making Rubyists more comfortable on Google Cloud Platform

Open Source Programs Office — Fri, 05 Aug 2016 17:00:00 +0000

One of the many open source efforts at Google is the Google Cloud Platform (GCP) native libraries for our most popular languages. One of these libraries is the gcloud-ruby project on GitHub which is released as the gcloud gem on rubygems.org. There are several gems for accessing Google Cloud Platform resources from Ruby but this gem is different. It is hand coded by Rubyists for Rubyists and that has some distinct advantages.

Many of us have had experience working with libraries that are clearly ported from another language. I usually talk about them as Ruby with a Java accent or Python with a Perl accent. Generally they work just fine but you can run into some low level friction — sometimes things just don’t feel right. Native gems written by members of the community solve this problem. In the case of gcloud-ruby there are some really concrete examples.

First, gcloud-ruby uses syntax that is similar to other popular Ruby libraries. For example, the syntax for specifying a table schema in BigQuery (Google Cloud Platform's very large scale data warehouse) looks like this:

table = dataset.create_table "baby_names" do |schema|
  schema.string "name"
  schema.string "sex"
  schema.integer "number"
end

Creating the same table in popular Ruby on Rails looks like this:

create_table "baby_names" do |schema|
  schema.string "name"
  schema.string "sex"
  schema.integer "number
end

The two are nearly identical. That makes getting up to speed on BigQuery easier and quicker than it would be if the Ruby library didn't use patterns that are already known to the majority of Rubyists.

Another way the gcloud-ruby library meets the community where it is at is by embracing the community's fondness for doing things several different ways. In Ruby there are often several correct ways to do a given task.

The gcloud-ruby library is no exception. There are a few different ways to authenticate and create the objects you use to interact with the API. Ruby also has many common methods that have aliases. In the standard library Enumerable#map and Enumerable#collect actually run the same code path for example. In gcloud-ruby the vision API uses aliases. Google Cloud Vision provides a single endpoint: annotate. gcloud-ruby has an annotate method but also aliases this method as mark and detect if those make more sense to you (detect is the method that makes the most sense to my brain so that's the one I use). By providing a couple of different aliases it can mean the first thing you try is more likely to work. This speeds up development time and makes learning the library easier.

The last way the gcloud-ruby gem makes Rubyists feel at home is by having comprehensive tests, a common value and popular discussion topic for the Ruby community. gcloud-ruby uses minitest-spec for testing, a popular choice that most Rubyists can easily read. When I was learning the storage API I looked at the tests for storage to learn how to use the library. There is outstanding documentation as well for those who prefer learning that way but I'm so used to looking at tests that I really appreciated that gcloud-ruby has well written and easily accessible tests.

Above are three examples of how hand-coded libraries from within the community can improve the user experience when learning to use tools. Of course, doing all the development on GitHub in the open also helps. Users can easily see what bugs people have run into and what features are next up in the production queue. And if a user has a feature request (like the previously mentioned Cloud Vision support) they can create a GitHub issue.

If you’re a Rubyist, give gcloud-ruby a shot and let us know what you think!

By Aja Hammerly, Developer Advocate

Stories from Google Code-in: KDE, MetaBrainz and Haiku

Open Source Programs Office — Mon, 01 Aug 2016 17:00:00 +0000

Google Code-in is our annual contest that gives students age 13 to 17 experience in computer science through contributions to open source projects. This blog post is the second installment in our series reflecting on the experiences of Google Code-in 2015 grand prize winners. Be sure to check out the first post in the series.

This week we profile three more grand prize winners from Google Code-in 2015. These students came from all around the world to celebrate with us in June after successfully completing 692 tasks that resulted in significant contributions to the participating open source projects.

Google Code-in 2015 Grand Prize Winners and Mentors were treated to a cruise around San Francisco Bay.

Students were paired with mentors who guided them as they learned both new technologies and how to collaborate on real-world projects. While most students had some programming experience, many were new to open source. In the end, they learned new skills, connected with open source communities and many will continue to contribute to open source projects.

We’re proud of all of the participants and grateful to the mentors who helped them. We invited the contest winners to write about their experience and many took us up on the offer. Here are their stories:

First up today is Imran Tatriev, a student from Kazakhstan who decided to work on the KDE project because loved their philosophy and had experience with C++ and Qt. He was a finalist in Google Code-in 2014 when he worked with the OpenMRS project.

Imran’s work on KDE included contributing to projects such as KDevelop, Marble and GCompris. His biggest challenge was working on the KDevelop IDE’s debugger where he was tasked with highlighting crashed threads. Highlighting the crashed thread was trivial, finding the thread that had crashed was not. It took him five days to solve that problem and he credits his mentor with helping him to work through it.

In the end, Imran learned a lot about regular expressions, the architecture of large software projects, C++ and unit testing. What did he like most about his Google Code-in experience? Imran writes: “The most valuable moments were meeting wonderful and smart people.” He plans to continue working with KDE and apply for Google Summer of Code.

Next is Caroline Gschwend, a student from the US who worked on the MetaBrainz project. Both of her parents are computer scientists and she credits them with spurring her interest.

A homeschool student with a unique approach to education, Caroline loves to learn and voraciously consumes free online resources. She had this to say: “I think that free, online learning is an amazing benefit to our society. With access to a computer and the internet, anyone, anywhere, can learn anything.”

Caroline discovered Google Code-in through her mother who had, in turn, discovered the contest through Google for Education. Caroline dug in and decided it was right up her alley. She loved that it embraced beginners with open arms and introduced new people to open source. Ultimately, she decided to work with MetaBrainz because, as a classically trained violinist, MusicBrainz piqued her interest. Their projects are primarily written in Perl and Python and, while Caroline was fluent in Java, it was too interesting to pass up.

As with most students, Caroline found collaboration to be a big part of the learning curve -- from GitHub to Git and IRC. Her mentors and other community contributors on IRC helped Caroline through the process and, looking back, she found that collaboration to be her favorite part of the whole experience. She loved that the mentors helped her to produce professional quality work rather than focusing on quantity.

Google Code-in gave Caroline a chance to learn about collaboration, Inkspace, icon design, web development and more. She has continued her work in open source and plans to apply for Google Summer of Code.

The last student we’re highlighting today is Vale Tolpegin, a student from the US who worked on the Haiku project, an open source operating system for personal computers. He also participated in Google Code-in 2014 but didn’t feel his skills were sharp enough to attack the more challenging tasks, like the ones he tackled this time around for Haiku.

Vale took on a wide range of tasks from documentation to application development, his favorite being the creation of the Haiku Hardware Repository. The repository is a Django website that lets people search and share hardware tests to determine if a given machine will work with Haiku.

He ran into a sticky issue early on, spending nearly a week finding a race condition within an application maintained by Haiku. Vale found it frustrating, but his mentors helped him see it through to the end. That wasn’t the only big challenge he ran into and, ultimately, bested: he spent another week debugging a Remote Desktop Application, software which had a very large code base.

Despite the two time consuming challenges, Vale managed to accomplish a lot more during the contest, including building a graph plotter and fixing bugs in the Haiku package manager. Vale had this to say:

“After finishing GCI, I have continued to work with Haiku and the experiences I have gained will continue to have an impact on me for years to come. Participating in GCI has truly been a life-changing experience!”

Thank you to Imran, Caroline and Vale for their contributions to open source and for sharing their Google Code-in experiences with us. Stay tuned, we’ve got two more posts coming in this series!

By Josh Simmons, Open Source Programs Office

Stories from Google Code-in: FOSSASIA and Haiku

Open Source Programs Office — Fri, 29 Jul 2016 17:11:00 +0000

Google Code-in is our annual contest to help pre-university students gain real-world computer science experience by taking on tasks of varying difficulty levels with the help of volunteer mentors. These tasks are created by open source projects so while learning, the students are contributing to the software many of us use on a daily basis.

The finalists and winners for our 2015/2016 season were announced in February and, in June, the grand prize winners joined us for four days of learning and celebration. Students and their guardians came from all around the world. One of my favorite things, as one of the Googler hosts, was seeing the light bulbs go on above parents’ heads as they came to understand open source and why it’s so important. These parents and guardians were even more proud of the students as they learned how much their teenager has contributed to the world through participating in Google Code-in.

We’ve invited contest winners and organizations to write about their experience and will be sharing their stories in a series of blog posts. This marks the first post in the series.

Google Code-in 2015 Grand Prize Winners and Mentors

Let’s start with Jason Wong, a student from the US who worked with FOSSASIA. FOSSASIA supports open source developers in Asia through events and coding programs.

Jason got into computer science during middle school at a summer camp where he built a website describing the differences between Linux, OS X, and Windows. He dove deeper into web development by learning PHP and JavaScript through YouTube videos. He enjoyed being able to build more complex and dynamic websites. Like many new developers, Jason became very confident but did not concern himself with important aspects of programming like testing.

He learned about Google Code-in when Stephanie Taylor, fellow open source program manager who manages the GCI program here at Google, gave a talk at his school. Jason dove right in picking FOSSASIA as the project he would contribute to.

FOSSASIA offered Jason a chance to learn a lot about development and open source. He worked on their event pages, integrated Loklak and added an RSS section to their website, gaining experience with version control, Docker, Pharo and Node.js in the process. Most importantly, Jason learned about collaboration. He had this to say:

“Collaboration is so important in the open source community as it allows everyone to come together to help the world. Google Code-in has persuaded me to contribute to open source in the future.”

Next up we have Hannah Pan, another US student. She chose to work on Haiku, an open source operating system built for personal computers, because it used the C/C++ language which she was already confident with.

Hannah got into computer science through a high school AP course and discovered Google Code-in through this blog (woohoo!). She decided to participate even though it had already been underway for two weeks. Aiming just to make the top 10 in order to have a chance at being a finalist (and earn a hoodie), Hannah finished as a grand prize winner!

The learning curve was steep: *nix commands, build tools and GitHub all presented new challenges. She was surprised how much code she had to sift through sometimes just to isolate the cause of minor bugs.

Like all of the participants, Hannah found her mentors to be crucial in providing both technical guidance and moral support. She explained, “I was amazed at my mentors’ expertise, dedication, modesty, and high standards. They taught me to strive for excellence rather than settle for mediocrity.”

Among other things, Hannah added localization support to the Tipster app, fixed extractDebugInfo, and even wrote a how-to article relating to the work. Reflecting on her experience, Hannah wrote:

“On the technical side, not only have I learned a lot, but I have realized how much more I have yet to learn. In addition, it has taught me some important life skills that no doubt will benefit me in my future endeavors. I’d like to thank my mentors and other students who inspired me and pushed me to do my best.”

Thank you to Jason and Hannah both for contributing to open source and sharing their Google Code-in experiences with us. Stay tuned as we continue this series in our next blog post!

By Josh Simmons, Open Source Programs Office

From Google Summer of Code to Game of Thrones on the Back of a JavaScript Dragon (Part 1)

Open Source Programs Office — Tue, 26 Jul 2016 21:00:00 +0000

This guest post is a part of a short series about Tatyana Goldberg and Guy Yachdav, instructors at Technical University of Munich, and the journey that was inspired by their participation as Google Summer of Code mentors for the BioJS project.

Hello there! We are from the BioJavaScript (BioJS) project which first joined Google Summer of Code (GSoC) in 2014. Our experience in the program set us on a grand open source adventure that we’ll be sharing with you in a series of blog posts. We hope you enjoy our story and, more importantly, hope it inspires you to pursue your own open source adventure.

Tatyana Goldberg and Guy Yachdav, GSoC mentors and open source enthusiasts. Photo taken at the MorpheusCup competition Luxembourg, May 2016.

We came together around the BioJS community, an open source project for creating beautiful and interactive open source visualizations of biological data on the web. BioJS visualizations are made up of components which have a modular design. This modular design enables several things: they can be used by non-programmers, they can be combined to make more complex visualizations, and they can be easily integrated into existing web applications. Despite being a young community, BioJS already has traction in industry and academia.

In early 2014 we decided to apply for GSoC and we were fortunate to have our application accepted on our first try. The experience was extremely positive — the five students we accepted delivered great software and they had a big impact on the BioJS community:

The number of mailing list subscribers doubled in less than a month.
All five of our accepted students from 2014 became core developers.
Students were invited to six international conferences to share their work.
Students helped organize the first BioJS conference held July 2015.
Most importantly, the students have independently designed BioJS version 2.0 which positioned BioJS as the leading open source visualization library for biological data.

You can see three examples of the work GSoC students did on BioJS below:

MSAViewer is a visualization and analysis of multiple sequence alignments and was developed by Sebastian Wilzbach. Proteome Viewer is a multilevel visualization of proteomes in the UniProt database and was developed by Jose Villaveces. Genetic Variation Viewer is visualization of the number and type of mutations at each position in a biological sequence and was developed by Saket Choudhary.

We learned a lot in the first year we participated in Google Summer of Code. Here are some of the takeaways that are especially relevant to mentors and organizations that are considering joining the program:

GSoC is a great source of dedicated and enthusiastic young developers.
Mentors need to carefully manage students, listen to them and let them lead initiatives when it makes sense.
Org admins should leverage success in GSoC beyond the program.
Orgs need to find the most motivated students and make sure their projects are feasible.
People want to share in your success, so participation in GSoC can start a positive feedback loop attracting new contributors and users.
Most importantly: the ideas behind GSoC - the love for open source and coding - are contagious and spread easily to larger audiences, especially to students and other people who work in academia. Just try it!

Our positive experience spurred us to seek out and conquer new challenges. Stay tuned for our next post where we explain how GSoC inspired us to create a popular new class and how we applied data science to Game of Thrones.

By Tatyana Goldberg and Guy Yachdav, BioJS and TU Munich

Omnitone: Spatial audio on the web

Open Source Programs Office — Mon, 25 Jul 2016 17:00:00 +0000

Spatial audio is a key element for an immersive virtual reality (VR) experience. By bringing spatial audio to the web, the browser can be transformed into a complete VR media player with incredible reach and engagement. That’s why the Chrome WebAudio team has created and is releasing the Omnitone project, an open source spatial audio renderer with the cross-browser support.

Our challenge was to introduce the audio spatialization technique called ambisonics so the user can hear the full-sphere surround sound on the browser. In order to achieve this, we implemented the ambisonic decoding with binaural rendering using web technology. There are several paths for introducing a new feature into the web platform, but we chose to use only the Web Audio API. In doing so, we can reach a larger audience with this cross-browser technology, and we can also avoid the lengthy standardization process for introducing a new Web Audio component. This is possible because the Web Audio API provides all the necessary building blocks for this audio spatialization technique.

Omnitone Audio Processing Diagram

The AmbiX format recording, which is the target of the Omnitone decoder, contains 4 channels of audio that are encoded using ambisonics, which can then be decoded into an arbitrary speaker setup. Instead of the actual speaker array, Omnitone uses 8 virtual speakers based on an the head-related transfer function (HRTF) convolution to render the final audio stream binaurally. This binaurally-rendered audio can convey a sense of space when it is heard through headphones.

The beauty of this mechanism lies in the sound-field rotation applied to the incoming spatial audio stream. The orientation sensor of a VR headset or a smartphone can be linked to Omnitone’s decoder to seamlessly rotate the entire sound field. The rest of the spatialization process will be handled automatically by Omnitone. A live demo can be found at the project landing page.

Throughout the project, we worked closely with the Google VR team for their VR audio expertise. Not only was their knowledge on the spatial audio a tremendous help for the project, but the collaboration also ensured identical audio spatialization across all of Google’s VR applications - both on the web and Android (e.g. Google VR SDK, YouTube Android app). The Spatial Media Specification and HRTF sets are great examples of the Google VR team’s efforts, and Omnitone is built on top of this specification and HRTF sets.

With emerging web-based VR projects like WebVR, Omnitone’s audio spatialization can play a critical role in a more immersive VR experience on the web. Web-based VR applications will also benefit from high-quality streaming spatial audio, as the Chrome Media team has recently added FOA compression to the open source audio codec Opus. More exciting things like VR view integration, higher-order ambisonics and mobile web support will also be coming soon to Omnitone.

We look forward to seeing what people do with Omnitone now that it's open source. Feel free to reach out to us or leave a comment with your thoughts and feedback on the issue tracker on GitHub.

By Hongchan Choi and Raymond Toy, Chrome Team

Due to the incomplete implementation of multichannel audio decoding on various browsers, Omnitone does not support mobile web at the time of writing.

Kubernetes 1.3 is here!

Open Source Programs Office — Thu, 21 Jul 2016 17:00:00 +0000

With all of the excitement being generated around the Kubernetes 1.3 release and the first anniversary of Kubernetes 1.0 (#k8sbday), now is a great time to point out some of the features that enterprise users should be taking note of.

If you’re not familiar with Kubernetes, let me get you up to speed.

Kubernetes is an open-source container automation framework that builds upon 15 years of experience of running production workloads at Google. Once you declare a desired state, Kubernetes works to drive your system toward that state. As a developer this means less time handling trivial tasks that a computer can automate and more time focusing on developing applications that provide value to users.

Additionally, Kubernetes aims to be a framework that you can operate at planetary scale, run anywhere, and never outgrow.

With the release of Kubernetes 1.3, Kubernetes is closer than ever to meeting those goals; the 1.3 release adds exciting features such as:

cross cluster federation allowing for production deployments to be spread across zones, regions, or even different cloud providers
local development support with the minikube tool
performance upgrades with up to 2,000 nodes with 60,000 containers running
better support for stateful applications
and much more

Aside from features, the coolest part about working with Kubernetes is hearing user stories. I’ll soon be publishing an interview with Joseph Jacks, co-founder of Kismatic, the enterprise Kubernetes company, on the Kubernetes blog.

Joseph is very active in the Kubernetes community and has extensive experience with Kubernetes in production. In the interview I ask him why he bet his business on Kubernetes, what could be better, and how he sees Kubernetes growing in the near future.

Kubernetes has many, many features to offer that I didn’t get to cover in this short write-up. If you know anyone that needs to ramp up on Kubernetes, the easiest way is the free course I created with Kelsey Hightower, Scalable Microservices with Kubernetes. The course covers the basic features of Kubernetes. If you want an overview of what’s new in Kubernetes 1.3, feel free to look at the “What’s new in Kubernetes 1.3” video or slides.

Finally for a more in-depth look at the 1.3 release, make sure to check out: 5 days of Kubernetes 1.3 blog series.

Want to learn more about container orchestration and cloud native platforms? Here’s some recommended reading to follow up with:

By Carter Morgan, Developer Programs Engineer

Announcing an Open Source ADC board for BeagleBone

Open Source Programs Office — Wed, 20 Jul 2016 17:00:00 +0000

Cross posted on the Google Research Blog
Working with electronics, we often find ourselves soldering up a half baked electronic circuit to detect some sort of signal. For example, last year we wanted to measure the strength of a carrier. We started with traditional analog circuits — amplifier, filter, envelope detector, threshold. You can see some of our prototypes in the image below; they get pretty messy.

While there's a certain satisfaction in taming a signal using the physical properties of capacitors, coils of wire and transistors, it's usually easier to digitize the signal with an Analog to Digital Converter (ADC) and manage it with Digital Signal Processing (DSP) instead of electronic parts. Tweaking software doesn't require a soldering iron, and lets us modify signals in ways that would require impossible analog circuits.

There are several standard solutions for digitizing a signal: connect a laptop to an oscilloscope or Data Acquisition System (DAQ) via USB or Ethernet, or use the onboard ADCs of a maker board like an Arduino. The former are sensitive and accurate, but also big and power hungry. The latter are cheap and tiny, but slower and have enough RAM for only milliseconds worth of high speed sample data.

That led us to investigate single board computers like the BeagleBone and Raspberry Pi, which are small and cheap like an Arduino, but have specs like a smartphone. And crucially, the BeagleBone's system-on-a-chip (SoC) combines a beefy ARMv7 CPU with two smaller Programmable Realtime Units (PRUs) that have access to all 512MB of system RAM. This lets us dedicate the PRUs to the time-sensitive and repetitive task of reading each sample out of an external ADC, while the main CPU lets us use the data with the GNU/Linux tools we're used to.

The result is an open source BeagleBone cape we've named PRUDAQ. It's built around the Analog Devices AD9201 ADC, which samples two inputs simultaneously at up to 20 megasamples per second, per channel. Simultaneous sampling and high sample rates make it useful for software-defined radio (SDR) and scientific applications where a built-in ADC isn't quite up to the task.

Our open source electrical design and sample code are available on GitHub, and GroupGets has boards ready to ship for $79. We also were fortunate to have help from Google intern Kumar Abhishek. He added support for PRUDAQ to his Google Summer of Code project BeagleLogic that performs much better than our sample code.

We started PRUDAQ for our own needs, but quickly realized that others might also find it useful. We're excited to get your feedback through the email list. Tell us what can be done with inexpensive fast ADCs paired with inexpensive fast CPUs!

Posted by Jason Holt, Software Engineer

Lessons from Professors’ Open Source Software Experience (POSSE) 2016

Open Source Programs Office — Wed, 06 Jul 2016 16:00:00 +0000

From Google Summer of Code to Google Code-in, the Open Source Programs Office does a lot to get students involved with open source. In order to learn more about supporting open source in academia, I attended the NSF funded Professors' Open Source Software Experience (POSSE) in Philadelphia. It was a great opportunity for us to better understand the challenges instructors face in weaving open source into their curriculum and hear solutions on how to bridge the gap.

Almost 30 university professors and community college lecturers attended the 3-day workshop. During the workshop, attendees worked in small groups getting hands on experience incorporating humanitarian free and open source software (HFOSS) into their teaching. Professors were able to talk, mingle and share best practices throughout the event.

The POSSE workshop is led by Heidi Ellis, Professor, Department of Computer Science and Information Technology at Western New England University, and Greg Hislop, Professor of Software Engineering and Senior Associate Dean for Academic Affairs at Drexel University. Heidi and Greg took over running POSSE five years after Red Hat began the program as an outreach effort to the higher education community. Red Hat continues as a collaborator in the effort. Around 40 university and community college professors participate in the program every year with over 100 individuals attending the workshop in the last four years.

Here are some of the challenges professors shared:

Very little guidance on how to bring FOSS into the classroom. No standard curriculum / syllabus available to reference.
Time investment required to change the curriculum.
Will not be rewarded for teaching FOSS courses.
Will not get funds to travel for workshops/conferences unless it’s to present a paper at a conference.
Many administrations aren’t aware that adding open source is beneficial for students since more and more companies use open source and expect their new hires to be familiar with it.

The next POSSE will be Nov 17-19. Faculty who are interested in attending POSSE, please click here to apply.

We also discussed a number of open source programs that are currently working to engage students with open source software development:

Corporate
Academic
- Rochester Institute of Technology: Free and Open Source Software and Free Culture Minor Program
- Rensselaer Polytechnic Institute: Rensselaer Center for Open Source
- Oregon State University: Open Source Lab
- University of California at Santa Cruz: Center for Research on Open Source Software

Thanks to Heidi, Greg and the FOSS2Serve team for organizing POSSE 2016! We look forward to taking what we’ve learned and using it to better support FOSS education in academia.

By Feiran Helen Hu, Open Source Programs Office

GitHub on BigQuery: Analyze all the code

Open Source Programs Office — Wed, 29 Jun 2016 20:06:00 +0000

Posted by Felipe Hoffa, Google Developer Advocate

Google, in collaboration with GitHub, is releasing an incredible new open dataset on Google BigQuery. So far you've been able to monitor and analyze GitHub's pulse since 2011 (thanks GitHub Archive project!) and today we're adding the perfect complement to this. What could you do if you had access to analyze all the open source software in the world, with just one SQL command?

The Google BigQuery Public Datasets program now offers a full snapshot of the content of more than 2.8 million open source GitHub repositories in BigQuery. Thanks to our new collaboration with GitHub, you'll have access to analyze the source code of almost 2 billion files with a simple (or complex) SQL query. This will open the doors to all kinds of new insights and advances that we're just beginning to envision.

For example, let's say you're the author of a popular open source library. Now you'll be able to find every open source project on GitHub that's using it. Even more, you'll be able to guide the future of your project by analyzing how it's being used, and improve your APIs based on what your users are actually doing with it.

On the security side, we've seen how the most popular open source projects benefit from having multiple eyes and hands working on them. This visibility helps projects get hardened and buggy code cleaned up. What if you could search for errors with similar patterns in every other open source project? Would you notify their authors and send them pull requests? Well, now you can. Some concepts to keep in mind while working with BigQuery and the GitHub contents dataset:

With BigQuery everyone gets a terabyte every month to run queries. If you've never tried BigQuery before, follow these getting started instructions.
The contents table has all the non-binary files in GitHub that are less than 1MB. It's a huge table, with more than 1.5 terabytes of data! This means the monthly terabyte for BigQuery queries won't last long if you want to query this table. To make your life easier, we've created extracts with only a sample of 10% of all files of the most popular projects, as well as another dataset with all the .go, .rb. .js, .php, .py, and .java code. Use them to make your free quota last!
If these tables are not enough, you can always create your own extracts (but you'll be billed for the respective storage). To do so, you could sign up for $300 in Google Cloud Platform credits. These credits could be used to store terabytes (and more) of data in BigQuery.
BigQuery makes it easy to join different datasets. How about ranking coding patterns by the number of stars their projects get? See a related post looking at the Hacker News effect on a project’s GitHub stars.
SQL is not enough? Learn how BigQuery allows you to run arbitrary JavaScript code inside SQL to enable a full range of possibilities.

To learn more, read GitHub's announcement and try some sample queries. Share your queries and findings in our reddit.com/r/bigquery and Hacker News posts. The ideas are endless, and I'll start collecting tips and links to other articles on this post on Medium.

Stay curious!

More statistics from Google Summer of Code 2016

Open Source Programs Office — Tue, 28 Jun 2016 18:49:00 +0000

Google Summer of Code (GSoC) 2016 is officially at its halfway point. Mentors and students have just completed their midterm evaluations and it’s time for our second stats post. This time we take a closer look at our participating students.

First, we’d like to highlight the universities with the most student participants. Congratulations are due to the International Institute of Information Technology - Hyderabad for claiming the top spot for the third consecutive year!

Country	School	2016 Accepted Students	2015 Accepted Students	12 Year Total
India	International Institute of Information Technology - Hyderabad	50	62	252
Sri Lanka	University of Moratuwa	29	44	320
Romania	University POLITEHNICA of Bucharest	24	14	155
India	Birla Institute of Technology and Science Pilani, Goa Campus	22	15	110
India	Birla Institute of Technology and Science, Pilani Campus	22	18	116
India	Indian Institute of Technology, Bombay	18	13	75
India	Indian Institute of Technology, Kharagpur	15	8	92
India	Indian Institute of Technology, Roorkee	15	8	57
India	Indraprastha Institute of Information Technology Delhi	15	7	27
India	Amrita School of Engineering, Amrita University, Amritapuri Campus	13	5	33
India	Indian Institute of Technology, Guwahati	13	5	38
Cameroon	University of Buea	12	10	26
India	Delhi Technological University	12	9	60
India	Indian Institute of Technology BHU Varanasi	12	12	37
Germany	TU Munich	11	7	45

Next, we are proud to announce that 2016 marks the largest number of female GSoC participants to date — 12% of accepted students are female, up 2.2% from 2015. This is good progress, but we are certain we can do better in the future to diversify our program. The Google Open Source team will continue our outreach to many organizations, for example, Grace Hopper and Black Girls Code, to increase this number even more 2017. If you have any suggestions of organizations we should work with, please let us know in the comments.

Finally, each year we like to look at the majors of students. As expected, the most common area of study for our participants is Computer Science (approximately 78%), but this year we have a wide variety of studies including Linguistics, Law, Music Technology and Psychology. The majority of our students this year are undergraduates (67%), followed by Masters (23%) and then PhD students (9%).

Although reviewing GSoC statistics each year is great fun, we want to stress that being “first place” is not the point of the program. Our goal is to get more and more students involved in creating free and open source software. We hope Google Summer of Code encourages contributions to projects that have the potential to make a difference worldwide. Congratulations to the students from all over the globe and keep up the good work!

By Mary Radomile, Open Source Programs Office

Announcing SyntaxNet: The World’s Most Accurate Parser Goes Open Source

Open Source Programs Office — Fri, 13 May 2016 19:08:00 +0000

Originally posted on the Google Research Blog

By Slav Petrov, Senior Staff Research Scientist

At Google, we spend a lot of time thinking about how computer systems can read and understand human language in order to process it in intelligent ways. Today, we are excited to share the fruits of our research with the broader community by releasing SyntaxNet, an open-source neural network framework implemented in TensorFlow that provides a foundation for Natural Language Understanding (NLU) systems. Our release includes all the code needed to train new SyntaxNet models on your own data, as well as Parsey McParseface, an English parser that we have trained for you and that you can use to analyze English text.

Parsey McParseface is built on powerful machine learning algorithms that learn to analyze the linguistic structure of language, and that can explain the functional role of each word in a given sentence. Because Parsey McParseface is the most accurate such model in the world, we hope that it will be useful to developers and researchers interested in automatic extraction of information, translation, and other core applications of NLU.

How does SyntaxNet work?

SyntaxNet is a framework for what’s known in academic circles as a syntactic parser, which is a key first component in many NLU systems. Given a sentence as input, it tags each word with a part-of-speech (POS) tag that describes the word's syntactic function, and it determines the syntactic relationships between words in the sentence, represented in the dependency parse tree. These syntactic relationships are directly related to the underlying meaning of the sentence in question. To take a very simple example, consider the following dependency tree for Alice saw Bob:

This structure encodes that Alice and Bob are nouns and saw is a verb. The main verb saw is the root of the sentence and Alice is the subject (nsubj) of saw, while Bob is its direct object (dobj). As expected, Parsey McParseface analyzes this sentence correctly, but also understands the following more complex example:

This structure again encodes the fact that Alice and Bob are the subject and object respectively of saw, in addition that Alice is modified by a relative clause with the verb reading, that saw is modified by the temporal modifier yesterday, and so on. The grammatical relationships encoded in dependency structures allow us to easily recover the answers to various questions, for example whom did Alice see?, who saw Bob?, what had Alice been reading about? or when did Alice see Bob?.

Why is Parsing So Hard For Computers to Get Right?

One of the main problems that makes parsing so challenging is that human languages show remarkable levels of ambiguity. It is not uncommon for moderate length sentences - say 20 or 30 words in length - to have hundreds, thousands, or even tens of thousands of possible syntactic structures. A natural language parser must somehow search through all of these alternatives, and find the most plausible structure given the context. As a very simple example, the sentence Alice drove down the street in her car has at least two possible dependency parses:

The first corresponds to the (correct) interpretation where Alice is driving in her car; the second corresponds to the (absurd, but possible) interpretation where the street is located in her car. The ambiguity arises because the preposition in can either modify drove or street; this example is an instance of what is called prepositional phrase attachment ambiguity.

Humans do a remarkable job of dealing with ambiguity, almost to the point where the problem is unnoticeable; the challenge is for computers to do the same. Multiple ambiguities such as these in longer sentences conspire to give a combinatorial explosion in the number of possible structures for a sentence. Usually the vast majority of these structures are wildly implausible, but are nevertheless possible and must be somehow discarded by a parser.

SyntaxNet applies neural networks to the ambiguity problem. An input sentence is processed from left to right, with dependencies between words being incrementally added as each word in the sentence is considered. At each point in processing many decisions may be possible—due to ambiguity—and a neural network gives scores for competing decisions based on their plausibility. For this reason, it is very important to use beam search in the model. Instead of simply taking the first-best decision at each point, multiple partial hypotheses are kept at each step, with hypotheses only being discarded when there are several other higher-ranked hypotheses under consideration. An example of a left-to-right sequence of decisions that produces a simple parse is shown below for the sentence I booked a ticket to Google.

Furthermore, as described in our paper, it is critical to tightly integrate learning and search in order to achieve the highest prediction accuracy. Parsey McParseface and other SyntaxNet models are some of the most complex networks that we have trained with the TensorFlow framework at Google. Given some data from the Google supported Universal Treebanks project, you can train a parsing model on your own machine.

So How Accurate is Parsey McParseface?

On a standard benchmark consisting of randomly drawn English newswire sentences (the 20 year old Penn Treebank), Parsey McParseface recovers individual dependencies between words with over 94% accuracy, beating our own previous state-of-the-art results, which were already better than any previous approach. While there are no explicit studies in the literature about human performance, we know from our in-house annotation projects that linguists trained for this task agree in 96-97% of the cases. This suggests that we are approaching human performance—but only on well-formed text. Sentences drawn from the web are a lot harder to analyze, as we learned from the Google WebTreebank (released in 2011). Parsey McParseface achieves just over 90% of parse accuracy on this dataset.

While the accuracy is not perfect, it’s certainly high enough to be useful in many applications. The major source of errors at this point are examples such as the prepositional phrase attachment ambiguity described above, which require real world knowledge (e.g. that a street is not likely to be located in a car) and deep contextual reasoning. Machine learning (and in particular, neural networks) have made significant progress in resolving these ambiguities. But our work is still cut out for us: we would like to develop methods that can learn world knowledge and enable equal understanding of natural language across all languages and contexts.

To get started, see the SyntaxNet code and download the Parsey McParseface parser model. Happy parsing from the main developers, Chris Alberti, David Weiss, Daniel Andor, Michael Collins & Slav Petrov.