Google Data

Posted by Maggie Johnson, Director of Education and University Relations

We have just completed another round of the Google Faculty Research Awards, our biannual open call for research proposals on Computer Science and related topics, including systems, machine perception, structured data, robotics, and mobile. Our grants cover tuition for a graduate student and provide both faculty and students the opportunity to work directly with Google researchers and engineers.

This round we received 808 proposals, an increase of 12% over last round, covering 55 countries on 6 continents. After expert reviews and committee discussions, we decided to fund 122 projects, with 20% of the funding awarded to universities outside the U.S. The subject areas that received the highest level of support were systems, human-computer interaction, and machine perception.

The Faculty Research Award program enables us to build strong relationships with faculty around the world who are pursuing innovative research, and plays an important role for Google’s Research organization by fostering an exchange of ideas that advances the state of the art. Each round, we receive proposals from faculty who may be just starting their careers, or who might be experimenting in new areas that help us look forward and innovate on what’s emerging in the CS community.

Congratulations to the well-deserving recipients of this round’s awards. If you are interested in applying for the next round (deadline is April 15), please visit our website for more information.

Posted by Miriam Schneider, Google for Education team

(Cross-posted from the Google for Education Blog)

Science is about observing and experimenting. It’s about exploring unanswered questions, solving problems through curiosity, learning as you go and always trying again.

That’s the spirit behind the fifth annual Google Science Fair, kicking off today. Together with LEGO Education, National Geographic, Scientific American and Virgin Galactic, we’re calling on all young researchers, explorers, builders, technologists and inventors to try something ambitious. Something imaginative, or maybe even unimaginable. Something that might just change the world around us.

From now through May 18, students around the world ages 13-18 can submit projects online across all scientific fields, from biology to computer science to anthropology and everything in between. Prizes include $100,000 in scholarships and classroom grants from Scientific American and Google, a National Geographic Expedition to the Galapagos, an opportunity to visit LEGO designers at their Denmark headquarters, and the chance to tour Virgin Galactic’s new spaceship at their Mojave Air and Spaceport. This year we’re also introducing an award to recognize an Inspiring Educator, as well as a Community Impact Award honoring a project that addresses an environmental or health challenge.

It’s only through trying something that we can get somewhere. Flashlights required batteries, then Ann Makosinski tried the heat of her hand. His grandfather would wander out of bed at night, until Kenneth Shinozuka tried a wearable sensor. The power supply was constantly unstable in her Indian village, so Harine Ravichandran tried to build a different kind of regulator. Previous Science Fair winners have blown us away with their ideas. Now it’s your turn.

Big ideas that have the potential to make a big impact often start from something small. Something that makes you curious. Something you love, you’re good at, and want to try.

So, what will you try?

Posted by Michael Rennaker, Google University Relations

In 2009, Google created the PhD Fellowship program to recognize and support outstanding graduate students doing exceptional work in Computer Science (CS) and related disciplines. In that time we’ve seen past recipients add depth and breadth to CS by developing new ideas and research directions, from building new intelligence models to changing the way in which we interact with computers to advancing into faculty positions, where they go on to train the next generation of researchers.

Reflecting our continuing commitment to building strong relations with the global academic community, we are excited to announce the latest North American Google PhD Fellows. The following 15 fellowship recipients were chosen from a highly competitive group, and represent the outstanding quality of nominees provided by our university partners:

• Justin Meza, Google US/Canada Fellowship in Systems Reliability (Carnegie Mellon University)
• Waleed Ammar, Google US/Canada Fellowship in Natural Language Processing (Carnegie Mellon University)
• Aaron Parks, Google US/Canada Fellowship in Mobile Networking (University of Washington)
• Kyle Rector, Google US/Canada Fellowship in Human Computer Interaction (University of Washington)
• Nick Arnosti, Google US/Canada Fellowship in Market Algorithms (Stanford University)
• Osbert Bastani, Google US/Canada Fellowship in Programming Languages (Stanford University)
• Carl Vondrick, Google US/Canada Fellowship in Machine Perception, (Massachusetts Institute of Technology)
• Wojciech Zaremba, Google US/Canada Fellowship in Machine Learning (New York University)
• Xiaolan Wang, Google US/Canada Fellowship in Structured Data (University of Massachusetts Amherst)
• Muhammad Naveed, Google US/Canada Fellowship in Security (University of Illinois at Urbana-Champaign)
• Masoud Moshref Javadi, Google US/Canada Fellowship in Computer Networking (University of Southern California)
• Riley Spahn, Google US/CanadaFellowship in Privacy (Columbia University)
• Saurabh Gupta, Google US/Canada Fellowship in Computer Vision (University of California, Berkeley)
• Yun Teng, Google US/Canada Fellowship in Computer Graphics (University of California, Santa Barbara)
• Tan Zhang, Google US/Canada Fellowship in Mobile Systems (University of Wisconsin-Madison)

This group of students represent the next generation of researchers who endeavor to solve some of the most interesting challenges in Computer Science. We offer our congratulations, and look forward to their future contributions to the research community with high expectations.

Posted by Walter Jetz, Dept. of Ecology and Evolutionary Biology, Yale University, and Dave Thau, Developer Advocate, Google Earth Engine, with support from Robert Guralnick, Dept. of Natural History, University of Florida

Nature reserves have a vital role for protecting biodiversity and its many functions. However, there is often insufficient information available to determine where to most effectively invest conservation efforts to prevent future extinctions, or which species may be left out of conservation actions entirely.

To help address these issues, Map of Life, in collaboration with Google Earth Engine, has now pre-released a new service to pinpoint at-risk species and where in the world that they occur. At the fingertips of regional naturalists, conservation groups, resource managers and global threat assessors, the tool has the potential to help identify and close key information gaps and highlight species of greatest concern.

Take the Tamaulipas Pygmy Owl, one of the smallest owls in the world that is restricted to highland forests in Mexico. The consensus range map for the species indicates a broad distribution of over 50,000 km²:

Left: Tamaulipas Pygmy Owl (Glaucidium sanchezi, photo credit: Adam Kent). Right: Map of Life consensus range map showing the potentially habitable range of this species.

But accounting for available habitat in the area using remotely sensed information presents a different picture: less than 10% of this range are forested and at the suitable elevation.

The owl’s potential protection is likely to occur in only around 1,000 km2 that are under formal protection, representing seven reserves of which only two have greater than 100 km2 area. This is much less than would be desirable for a species with this small a global range.

Another species example, the Hildegard’s Tomb Bat, is similarly concerning: less than 6,000 km² of suitable range remains for this forest specialist in East Africa, with less than half currently under protection.

A demonstration of this tool for 15 example species was pre-released at the decadal World Parks Congress in Sydney Australia last November to the global community of conservation scientists and practitioners. In the coming months this interactive evaluation will be expanded to thousands more species, providing a valuable resource to aid in global conservation efforts. For more information and updates, follow Map of Life.

Posted by Maggie Johnson, Director of Education and University Relations

Last July, Google and the Institute of Electrical and Electronics Engineers Power Electronics Society (IEEE PELS) announced the Little Box Challenge, a competition designed to push the forefront of new technologies in the research and development of small, high power density inverters.

In parallel, we announced the Little Box Challenge award program designed to help support academics pursuing groundbreaking research in the area of increasing the power density for DC-­to­-AC power conversion. We received over 100 proposals and today we are proud to announce the following recipients of the academic awards:

The recipients hail from many different parts of the world and were chosen based on their very strong and thoughtful entries dealing with all the issues raised in the request for proposals. Each of these researchers will receive approximately $30,000 US to support their research into high power density inverters, and are encouraged to use this work to attempt to win the $1,000,000 US grand prize for the Little Box Challenge.

There were many submissions beyond those chosen here that reviewers also considered to be very promising. We encourage all those who did not receive funding to still participate in the Little Box Challenge, and pursue improvements not only in power density, but also in the reliability, efficiency, safety, and cost of inverters (and of course, to attempt to win the grand prize!)

Posted Vint Cerf, Chief Internet Evangelist, Roy Want and Max Senges, Google Research

Imagine a world in which access to networked technology defies the constraints of desktops, laptops or smartphones. A future where we work seamlessly with connected systems, services, devices and “things” to support work practices, education, and daily interactions. While the Internet of Things (IoT) conjures a vision of “anytime, any place” connectivity for all things, the realization is complex given the need to work across interconnected and heterogeneous systems, and the special considerations needed for security, privacy, and safety.

Google is excited about the opportunities the IoT presents for future products and services. To further the development of open standards, facilitate ease of use, and ensure that privacy and security are fundamental values throughout the evolution of the field, we are in the process of establishing an open innovation and research program around the IoT. We plan to bring together a community of academics, Google experts and potentially other parties to pursue an open and shared mission in this area.

As a first step, we are announcing an open call for research proposals for the Open Web of Things:

• Researchers interested in the Expedition Lead Grant should build a team of PIs and put forward a proposal outlining a draft research roadmap both for their team(s), as well as how they propose to integrate related research that is implemented outside their labs (e.g., Individual Project Grants).
• For the Individual Project Grants we are seeking research proposals relating to the IoT in the following areas (1) user interface and application development, (2) privacy & security, and (3) systems & protocols research.

Importantly, we are open to new and unorthodox solutions in all three of these areas, for example, novel interactions, usable security models, and new approaches for open standards and evolution of protocols.

Additionally, to facilitate hands-on research supporting our mission driven research, we plan to provide participating faculty access to hardware, software and systems from Google. We look forward to your submission by January 21, 2015 and expect to select proposals early Spring. Selected PIs will be invited to participate in a kick-off workshop at Google shortly after.

Posted by Michel Benard, University Relations Manager and Cova Soto, Product Marketing Manager, Business Marketing Madrid

According to Eurostat data, over 5 million people under age 25 are currently out of work in Europe, in contrast to an increasing demand for people with digital skills such as Digital Marketing, Big Data, Ecommerce, Mobile App Development and Cloud Computing. In particular, Spanish employers are finding it difficult to find individuals with the right skills, due to the lack of the digital education available.

In an effort to make contributions towards solving Spain’s unemployment in this sector, Google Spain, the Spanish Ministry of Industry through their business school EOI, Universidad Complutense de Madrid and Interactive Advertising Bureau (IAB) are collaborating to build Google Activate, a series of massive open online courses (MOOCs) dedicated to teach digital skills to the young unemployed people in Spain. This is an example of how online education can be scaled to address educational and economic issues.

The inspiration for Google Activate began with the summer 2012 launch of Course Builder, an experimental platform developed on Google technologies designed to provide the capability for anyone to create an online environment that can be used for a wide variety of education-related activities. In September of that same year, Course Builder was made available in Europe, as part of the Google Faculty Summit in London.

Among the early adopters of Course Builder in Europe was a partnership that included the University of Alicante, who in October 2012 launched Unimooc Aemprende, a MOOC for entrepreneurs. This is just one example of the use of Course Builder to build a MOOC designed to solve a broad problem, in this case the acquisition of skills for launching a small business. More than 30,000 people have participated in Unimooc since its launch.

As of today, more than 148,000 people have registered for Activate with 13% of participants earning a certificate, which is obtained after 13 exams certified by either the EOI, Universidad Complutense de Madrid or or the IAB (Interactive Advertising Bureau). Such certificates are being used by the awardees in their LinkedIn profile to position themselves for a job in the digital economy, where many jobs are being created. More than 19,000 students are already certified in one of the 5 digital areas.

Google Activate has plans to increase the number of students with digital skills reaching 160,000 with plans to expand further to other countries in the world.

Posted by Maggie Johnson, Director of Education and University RelationsRecently, Tsinghua University and Google collaborated to host the 2014 APAC MOOC Focused Faculty Workshop in Shanghai, China. The workshop brought together 37 professors from 12 co…

Google Engineers Christian Szegedy, Scott Reed^*, Dumitru Erhan, and Dragomir Anguelov

Update – 26/02/2015
We recently discovered a bug in the evaluation methodology of our object detector. Consequently, the large numbers we initially reported below are not realistic, due to the fact that our separately trained context extractor was contaminated with half of the validation set images. Therefore, our initial results were overly optimistic and were not attainable by the methodology described in the paper. Re-evaluating our initial results, we have restricted ourselves to reporting only the single-model results on the other half of the dedicated validation set without retraining the models. With the updated evaluation, we are still able to report the best single-model result on the ILSVRC 2014 detection challenge data set, with 0.43 mAP when combining both Selective Search and MultiBox proposals with our post-classification model. The original draft of our paper “Scalable, High Quality Object Detection” has been updated to reflect this information. We are deeply sorry if our initial reported results caused any confusion in the community. Original post follows below. 
-C. Szegedy, S. Reed, D. Erhan, and D. Anguelov

The ILSVRC detection challenge is an influential academic benchmark for measuring the quality of object detection. This summer, the GoogLeNet team reported top results in the 2014 edition of the challenge, with ~2X improvement over the previous year’s best results. However, the quality of our results came at a high computational cost: processing each image took about two minutes on a state-of-the-art workstation.

Naturally, we began to think of how we could both improve the accuracy and reduce the computation time needed. Given the already high quality of previous results like those of GoogLeNet^[6], we expected that further improvements to detection quality would be increasingly hard to achieve. In our recent paper Scalable, High Quality Object Detection^[7], we detail advances that instead have resulted in an accelerated rate of progress in object detection:

Evolution of detection quality over time. On the y axis is the mean average precision of the best published results at any given time. The blue line shows result using individual models, the red line is multi-model ensembles. Overfeat[8] was the state-of-the-art at end of last year, followed by R-CNN[1] published in May. The later measurement points are the results of our team.[6,7]

As seen in the plot above, the mean average precision has been improved since August from 0.45 to 0.56: a 23% relative gain. The new approach can also match the quality of the former best solution with 140X reduced computational resources.

Most current approaches for object detection employ two phases^[1]: in the first phase, some hand-engineered algorithm proposes regions of interest in the image. In the second phase, each proposed region is run through a deep neural network, identifying which proposed patches correspond to an object (and what that object is).

For the first phase, the common wisdom^[1,2,3,4] was that it took skillfully crafted code to produce high quality region proposals. This has come with a drawback though: these methods don’t produce reliable scoring for the proposed regions. This forces the second phase to evaluate most of the proposed patches in order to achieve good results.

So we revisited our prior “MultiBox” work^[5], in which we let the computer learn to pick the proposals to see whether we could avoid relying on any of the hand-crafted methods above. Although the MultiBox method, using previous generation vision network architectures, could not compete with hand-engineered proposal approaches, there were several advantages of fully relying on machine learning only. First, the quality of proposals increases with each new improved network architecture or training methodology without additional programming effort. Second, the regions come with confidence scores which are used for trading off running time versus quality. Additionally, the implementation is simplified.

Once we used new variants of the network architecture introduced in [6], MultiBox also started to perform much better; Now, we could match the coverage of alternative methods with half as many proposal patches. Also, we changed our networks to take the context of objects into account, fueling additional quality gains for the second phase. Furthermore, we came up with a new way to train deep networks to learn more robustly even when some objects are not annotated in the training set, which improved both phases.

Besides the significant gains in mean average precision, we can now cut the number of evaluated patches dramatically at a modest loss of quality: the task that used to take 2 minutes of processing time for a single image on a workstation by the GoogLeNet ensemble (of 6 networks), is now performed under a second using a single network without using GPUs. If we constrain ourselves to a single category like “dog”, we can now process 50 images/second on the same machine by a more streamlined approach^[7] that skips the proposal generation step altogether.

As a core area of research in computer vision, object detection is used for providing strong signals for photo and video search, while high quality detection could prove useful for self-driving cars and automatically generated image captions. We look forward to the continuing research in this field.

References:

[1]  Rich feature hierarchies for accurate object detection and semantic segmentation
by Ross Girshick and Jeff Donahue and Trevor Darrell and Jitendra Malik (CVPR, 2014)

[2]  Prime Object Proposals with Randomized Prim’s Algorithm
by Santiago Manen, Matthieu Guillaumin and Luc Van Gool

[3]  Edge boxes: Locating object proposals from edges
by Lawrence C Zitnick, and Piotr Dollàr (ECCV 2014)

[4]  BING: Binarized normed gradients for objectness estimation at 300fps
by Ming-Ming Cheng, Ziming Zhang, Wen-Yan Lin and Philip Torr (CVPR 2014)

[5]  Scalable Object Detection using Deep Neural Networks
by Dumitru Erhan, Christian Szegedy, Alexander Toshev, and Dragomir Anguelov

[6]  Going deeper with convolutions
by Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke and Andrew Rabinovich

[7]  Scalable, high quality object detection
by Christian Szegedy, Scott Reed, Dumitru Erhan and Dragomir Anguelov

[8]  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Network by Pierre Sermanet, David Eigen, Xiang Zhang, Michael Mathieu, Rob Fergus and Yann LeCun

* A PhD student at University of Michigan — Ann Arbor and Software Engineering Intern at Google^↩

Posted by Beryl Nelson, Software Engineering Manager

By becoming more conscious of our own stereotypes and biases, and making use of the insights revealed by the research on bias and stereotype threat, unconscious decision making, and cognitive illusions, each of us can bring more to our work and create diverse, innovative, and meaningful organizations.

Since 2009, I’ve been reading literature about the challenges and successes in making diverse teams effective, and speaking about this research. My goal is to help everyone understand more about unconscious decision-making and other barriers to inclusion, and through knowledge, combat these effects.

A short summary:

• A team that is heterogeneous in meaningful ways is good for innovation, and good for business.
• There are many challenges to making such teams effective, such as unconscious decision making, stereotype threat, and other cognitive illusions.
• There is repeatable quantitative research which shows ways to combat some of these effects.
• The barriers to effectiveness may seem overwhelming, but there is hope! Meaningful change is possible, and some examples of successful change are cited below.

In a bit more detail:

1. Diversity is good for innovation and business. There is a correlation between financial success and the diversity of leadership teams, as shown in research by Catalyst, McKinsey and Cedric Herring. Further, research shows a strong correlation between having women on teams and innovation; concluding that there is a strong correlation between the presence of women and the social skills required to get ideas percolating into the open.
2. We all make decisions unconsciously, influenced by our implicit associations. As an example of these effects, a large proportion of CEOs are taller than the average population and height is strongly correlated with financial and career success. It’s long been argued that women and underrepresented minorities are not represented in CEO leadership because there aren’t enough qualified individuals in the labor pool. This “pipeline issue” argument can’t be made for short and average-height people, however. Simple, repeatable tests measure, via response time and error rate, the implicit associations we have between concepts. These associations are created as an adaptive response, but we must understand our own implicit biases in order to make better decisions.
3. Stereotype threat plays a role in preventing people from being fully effective. The low representation of women and minorities in Science has long been the source of a troubling question: is this an indication of a difference in innate ability (see Ben Barre’s response to Lawrence Summers’ remarks), or the result of some other effect? Claude Steele and his colleagues elegantly showed that two groups of people can have similar or opposite reactions, depending on the way a situation is presented. These and other experiments show that stereotype threat can compromise the performance of the subject of a stereotype, if he or she knows about the stereotype and cares about it.
4. Change is possible. The above and other challenges may make it seem nearly impossible to create a diverse and highly functioning organization, but dramatic change can be made. Take, for example, the discovery of biased decision making and effective changes made via the use of data in the MIT Science Faculty Study, or the amazing changes at Harvey Mudd college, which not only increased participation of women as Computer Science majors from 12% to 40% in five years, but also increased the total number of CS majors from 25 to 30 per year to 70 CS graduates in the class of 2014.

If you’re interested in learning more, watch the video about the data on diversity below. You can read the full research in the November issue of Communications of the Association of Computing Machinery. You can read even more using the full bibliography.

Posted by Kevin Murphy, Research Scientist and David Harper, Head of University Relations, EMEA

While the availability and size of data sets across a wide range of sources, from medical to scientific to commercial, continues to grow, there are relatively few people trained in the statistical and machine learning methods required to test hypotheses, make predictions, and otherwise create interpretable knowledge from this data. But what if one could automatically discover human-interpretable trends in data in an unsupervised way, and then summarize these trends in textual and/or visual form?

To help make progress in this area, Professor Zoubin Ghahramani and his group at the University of Cambridge received a Google Focused Research Award in support of The Automatic Statistician project, which aims to build an “artificial intelligence for data science”.

So far, the project has mostly been focussing on finding trends in time series data. For example, suppose we measure the levels of solar irradiance over time, as shown in this plot:This time series clearly exhibits several sources of variation: it is approximately periodic (with a period of about 11 years, known as the Schwabe cycle), but with notably low levels of activity in the late 1600s. It would be useful to automatically discover these kinds of regularities (as well as irregularities), to help further basic scientific understanding, as well as to help make more accurate forecasts in the future.

We can model such data using non-parametric statistical models based on Gaussian processes. Such methods require the specification of a kernel function which characterizes the nature of the underlying function that can accurately model the data (e.g., is it periodic? is it smooth? is it monotonic?). While the parameters of this kernel function are estimated from data, the form of the kernel itself is typically specified by hand, and relies on the knowledge and experience of a trained data scientist.

Prof Ghahramani’s group has developed an algorithm that can automatically discover a good kernel, by searching through an open-ended space of sums and products of kernels as well as other compositional operations. After model selection and fitting, the Automatic Statistician translates each kernel into a text description describing the main trends in the data in an easy-to-understand form.

The compositional structure of the space of statistical models neatly maps onto compositionally constructed sentences allowing for the automatic description of the statistical models produced by any kernel. For example, in a product of kernels, one kernel can be mapped to a standard noun phrase (e.g. ‘a periodic function’) and the other kernels to appropriate modifiers of this noun phrase (e.g. ‘whose shape changes smoothly’, ‘with growing amplitude’). The end result is an automatically generated 5-15 page report describing the patterns in the data with figures and tables supporting the main claims. Here is an extract of the report produced by their system for the solar irradiance data:

Extract of the report for the solar irradiance data, automatically generated by the automatic statistician.

The Automatic Statistician is currently being generalized to find patterns in other kinds of data, such as multidimensional regression problems, and relational databases. A web-based demo of a simplified version of the system was launched in August 2014. It allowed a user to upload a dataset, and to receive an automatically produced analysis after a few minutes. An expanded version of the service will be launched in early 2015 (we will post details when available). We believe this will have many applications for anyone interested in Data Science.

Posted by Shakir Mohamed and Charles Blundell, Google DeepMind, London

At Google, we continually explore and develop large-scale machine learning systems to improve our user’s experience, such as providing better video recommendations, deciding on the best language translation in a given context, or improving the accuracy of image search results. The data used to train these systems often contains many inconsistencies and missing elements, making progress towards large-scale probabilistic models designed to address these problems an important and ongoing part of our research. One principled and efficient approach for developing such models relies on an approach known as Variational Inference.

A renewed interest and several recent advances in variational inference^1,2,3,4,5,6 has motivated us to support and co-organise this year’s workshop on Advances in Variational Inference as part of the Neural Information Processing Systems (NIPS) conference in Montreal. These advances include new methods for scalability using stochastic gradient methods, the ability to handle data that arrives continuously as a stream, inference in non-linear time-series models, principled regularisation in deep neural networks, and inference-based decision making in reinforcement learning, amongst others.

Whilst variational methods have clearly emerged as a leading approach for tractable, large-scale probabilistic inference, there remain important trade-offs in speed, accuracy, simplicity and applicability between variational and other approximative schemes. The goal of the workshop will be to contextualise these developments and address some of the many unanswered questions through:

• Contributed talks from 6 speakers who are leading the resurgence of variational inference, and shaping the debate on topics of stochastic optimisation, deep learning, Bayesian non-parametrics, and theory.
• 34 contributed papers covering significant advances in methodology, theory and applications including efficient optimisation, streaming data analysis, submodularity, non-parametric modelling and message passing.
• A panel discussion with leading researchers in the field that will further interrogate these ideas. Our panelists are David Blei, Neil Lawrence, Shinichi Nakajima and Matthias Seeger.

The workshop presents a fantastic opportunity to discuss the opportunities and obstacles facing the wider adoption of variational methods. The workshop will be held on the 13th December 2014 at the Montreal Convention and Exhibition Centre. For more details see: www.variationalinference.org.

References:

1. Rezende, Danilo J., Shakir Mohamed, and Daan Wierstra, Stochastic Backpropagation and Approximate Inference in Deep Generative Models, Proceedings of the 31st International Conference on Machine Learning (ICML-14), 2014.

2. Gregor, Karol, Ivo Danihelka, Andriy Mnih, Charles Blundell and Daan Wierstra, Deep AutoRegressive Networks, Proceedings of the 31st International Conference on Machine Learning (ICML-14), 2014.

3. Mnih, Andriy, and Karol Gregor, Neural Variational Inference and Learning in Belief Networks, Proceedings of the 31st International Conference on Machine Learning (ICML-14), 2014.

4. Kingma, D. P. and Welling, M., Auto-Encoding Variational Bayes, Proceedings of the International Conference on Learning Representations (ICLR), 2014.

5. Broderick, T., Boyd, N., Wibisono, A., Wilson, A. C., & Jordan, M., Streaming Variational Bayes, Advances in Neural Information Processing Systems (pp. 1727-1735), 2013.

6. Hoffman, M., Blei, D. M., Wang, C., and Paisley, J., Stochastic Variational Inference, Journal of Machine Learning Research, 14:1303–1347, 2013.

Posted by Google Research Scientists Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan

“Two pizzas sitting on top of a stove top oven”
“A group of people shopping at an outdoor market”
“Best seats in the house”

People can summarize a complex scene in a few words without thinking twice. It’s much more difficult for computers. But we’ve just gotten a bit closer — we’ve developed a machine-learning system that can automatically produce captions (like the three above) to accurately describe images the first time it sees them. This kind of system could eventually help visually impaired people understand pictures, provide alternate text for images in parts of the world where mobile connections are slow, and make it easier for everyone to search on Google for images.

Recent research has greatly improved object detection, classification, and labeling. But accurately describing a complex scene requires a deeper representation of what’s going on in the scene, capturing how the various objects relate to one another and translating it all into natural-sounding language.

Automatically captioned: “Two pizzas sitting on top of a stove top oven”

Many efforts to construct computer-generated natural descriptions of images propose combining current state-of-the-art techniques in both computer vision and natural language processing to form a complete image description approach. But what if we instead merged recent computer vision and language models into a single jointly trained system, taking an image and directly producing a human readable sequence of words to describe it?

This idea comes from recent advances in machine translation between languages, where a Recurrent Neural Network (RNN) transforms, say, a French sentence into a vector representation, and a second RNN uses that vector representation to generate a target sentence in German.

Now, what if we replaced that first RNN and its input words with a deep Convolutional Neural Network (CNN) trained to classify objects in images? Normally, the CNN’s last layer is used in a final Softmax among known classes of objects, assigning a probability that each object might be in the image. But if we remove that final layer, we can instead feed the CNN’s rich encoding of the image into a RNN designed to produce phrases. We can then train the whole system directly on images and their captions, so it maximizes the likelihood that descriptions it produces best match the training descriptions for each image.

The model combines a vision CNN with a language-generating RNN so it can take in an image and generate a fitting natural-language caption.

Our experiments with this system on several openly published datasets, including Pascal, Flickr8k, Flickr30k and SBU, show how robust the qualitative results are — the generated sentences are quite reasonable. It also performs well in quantitative evaluations with the Bilingual Evaluation Understudy (BLEU), a metric used in machine translation to evaluate the quality of generated sentences.

A selection of evaluation results, grouped by human rating.

A picture may be worth a thousand words, but sometimes it’s the words that are most useful — so it’s important we figure out ways to translate from images to words automatically and accurately. As the datasets suited to learning image descriptions grow and mature, so will the performance of end-to-end approaches like this. We look forward to continuing developments in systems that can read images and generate good natural-language descriptions. To get more details about the framework used to generate descriptions from images, as well as the model evaluation, read the full paper here.

Posted by Dave Thau, Developer Advocate for Google Earth Engine and Karin Tuxen-Bettman, Program Manager, Google Earth Outreach

(Cross posted on the Official Google Australia Blog)

This week, thousands of people from more than 160 countries will gather in Sydney for the once-in-a-decade IUCN World Parks Congress to discuss the governance and management of protected areas. The Google Earth Outreach and Google Earth Engine teams will be at the event to showcase exemplars of how technology can help protect our environment.

Here are a few of the workshops and events happening in Sydney this week:

• Monday, November 10th – Tuesday, November 11th: Over the last couple of days, the Google Earth Outreach and Earth Engine teams delivered a 2-day hands-on workshop to develop the technical capacity of park managers, researchers, and communities. At this workshop, participants were introduced to Google mapping tools to help them with their conservation programs. 
• November 13 – 19: Google will be at the Oceans Pavilion inside the World Parks Congress to demonstrate how Trekker, Street View and Open Data Kit on Android mobile devices can assist with parks monitoring and management. 
• Friday, November 14, 9:30-10:30am: Join a Live Sydney Seahorse Hunt in Sydney Harbour, via Google Hangout, with Catlin Seaview Survey and Sydney Institute of Marine Science. Richard Vevers, Director of the Catlin Seaview Survey, will venture underwater to his favorite dive site and talk with experts about the unique marine life (including seahorses!) that explorers can expect to find around Sydney. Tune in here at 10:30am to catch all the action. 
• Saturday, November 15th, 8:30am: Networking for nature: the future is cool. Hear about how technology-driven ocean initiatives can help us better understand and strengthen our connection with our natural environments. WPCA-Marine’s plenary session will includes presentations by Sylvia Earle and Mission Blue, Catlin Seaview Survey, Google, Oceana, and SkyTruth. The session will also feature leading young marine professionals Mariasole Bianco and Rebecca Koss. 
• Saturday, November 15th, 12:15pm: We’ll be hosting a panel discussion on using Global Forest Watch to monitor protected areas in near-real-time. Global Forest Watch is a dynamic online alert system to help park rangers monitor and preserve vast stretches of parkland.
• Saturday, November 15th, 1:30 – 3:00pm: At the Biodiversity Pavilion join Walter Jetz from Yale and Dave Thau from Google for a presentation on Google Earth Engine and The Map of Life. The presentation will showcase how Google Earth Engine is being used in a variety of conservations efforts – including monitoring water resources, the health of the world’s forests, and measuring the impact of protected areas on biodiversity preservation. We will also announce a new global resource from The Map of Life for mapping and monitoring biodiverse ecosystems. 

We believe that technology can help address some of our world’s most pressing environmental challenges and we look forward to working with Australian conservationists to integrate technology into their work.

You can find us at the Oceans Pavilion inside the World Parks Congress, where we will be joined by our environmental partners including The Jane Goodall Institute, The World Resources Institute and The Map of Life.

We hope to see you at one of our events this week!

Posted by Alfred Spector, Vice President, Engineering

Recently, at the 27th ACM User Interface Software and Technology Symposium (UIST’14), Google Senior Research Scientist Shumin Zhai and University of Cambridge Lecturer Per Ola Kristensson received the 2014 Lasting Impact Award for their seminal paper SHARK²: a large vocabulary shorthand writing system for pen-based computers. Most simply put, this is one of those rare works that is responsible for fundamental and lasting advances in the industry, and is the basis for the rapidly growing number of keyboards that use gesture typing, including products such as ShapeWriter, Swype, SwiftKey, SlideIT, TouchPal, and Google Keyboard.

First presented 10 years ago at UIST’04, Shumin and Per Ola’s paper is a pioneering work on word-gesture keyboard interaction that described the architecture, algorithms and interfaces of a high-capacity multi-channel gesture recognition system-SHARK². SHARK² increased recognition accuracy and relaxed precision requirements by using the shape and location of gestures in addition to context based language models. In doing so, Shumin and Per Ola delivered a paradigm of touch screen gesture typing as an efficient method for text entry that has continued to drive the development of mobile text entry across the industry.

“Awarded for its scientific contribution of algorithms, insights, and user interface considerations essential to the practical realization of large-vocabulary shape-writing systems for graphical keyboards, laying the groundwork for new research, industrial applications, and widespread user benefit.”

Prior to joining Google in 2011, Shumin worked at the IBM Almaden Research Center for 15 years, where he originated and led the SHARK project, further developing and refining it to include a low latency recognition engine that introduced the ability to accurately recognize a large vocabulary of words based upon the patterns (sokgraphs) drawn on a touchscreen device. SHARK and SHARK² subsequently continued further development as ShapeWriter. During his tenure at IBM, Shumin additionally pursued a wide variety of HCI research areas including, but not limited to, studying the ease and efficiency of HCI interfaces, camera phone based motion sensing, and cross-device user experience.

At Google, Shumin has continued to inspire the Human-Computer Interaction research community, publishing prolifically and leading a group that incorporates HCI research, machine learning, statistical language modeling and mobile computing to advance the state of the art of text input for smart touchscreen keyboards. Building on his earlier work with SHARK/ShapeWriter, Gesture Typing is just one of the innovations that make things like typing messages on mobile device easier for hundreds of millions of people each day, and remains one of the most prominent features on Android keyboards.

Shumin has been highly active in academia during his career, as both visiting professor and lecturer at world-class universities, and is currently the Editor-in-Chief of ACM Transactions on Computer- Interaction, a Fellow of the ACM and a Member of the CHI Academy. We’re proud to congratulate Shumin and Per Ola on receiving one of the most prestigious honors in the Human-Computer Interaction (HCI) research community, and look forward to their future contributions.