Google Data

Posted by Quoc V. Le & Mike Schuster, Research Scientists, Google Brain Team

Ten years ago, we announced the launch of Google Translate, together with the use of Phrase-Based Machine Translation as the key algorithm behind this service. Since then, rapid advances in machine intelligence have improved our speech recognition and image recognition capabilities, but improving machine translation remains a challenging goal.

Today we announce the Google Neural Machine Translation system (GNMT), which utilizes state-of-the-art training techniques to achieve the largest improvements to date for machine translation quality. Our full research results are described in a new technical report we are releasing today: “Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation” [1].

A few years ago we started using Recurrent Neural Networks (RNNs) to directly learn the mapping between an input sequence (e.g. a sentence in one language) to an output sequence (that same sentence in another language) [2]. Whereas Phrase-Based Machine Translation (PBMT) breaks an input sentence into words and phrases to be translated largely independently, Neural Machine Translation (NMT) considers the entire input sentence as a unit for translation.The advantage of this approach is that it requires fewer engineering design choices than previous Phrase-Based translation systems. When it first came out, NMT showed equivalent accuracy with existing Phrase-Based translation systems on modest-sized public benchmark data sets.

Since then, researchers have proposed many techniques to improve NMT, including work on handling rare words by mimicking an external alignment model [3], using attention to align input words and output words [4] and breaking words into smaller units to cope with rare words [5,6]. Despite these improvements, NMT wasn’t fast or accurate enough to be used in a production system, such as Google Translate. Our new paper [1] describes how we overcame the many challenges to make NMT work on very large data sets and built a system that is sufficiently fast and accurate enough to provide better translations for Google’s users and services.

Data from side-by-side evaluations, where human raters compare the quality of translations for a given source sentence. Scores range from 0 to 6, with 0 meaning “completely nonsense translation”, and 6 meaning “perfect translation.”

The following visualization shows the progression of GNMT as it translates a Chinese sentence to English. First, the network encodes the Chinese words as a list of vectors, where each vector represents the meaning of all words read so far (“Encoder”). Once the entire sentence is read, the decoder begins, generating the English sentence one word at a time (“Decoder”). To generate the translated word at each step, the decoder pays attention to a weighted distribution over the encoded Chinese vectors most relevant to generate the English word (“Attention”; the blue link transparency represents how much the decoder pays attention to an encoded word).

Using human-rated side-by-side comparison as a metric, the GNMT system produces translations that are vastly improved compared to the previous phrase-based production system. GNMT reduces translation errors by more than 55%-85% on several major language pairs measured on sampled sentences from Wikipedia and news websites with the help of bilingual human raters.

An example of a translation produced by our system for an input sentence sampled from a news site. Go here for more examples of translations for input sentences sampled randomly from news sites and books.

In addition to releasing this research paper today, we are announcing the launch of GNMT in production on a notoriously difficult language pair: Chinese to English. The Google Translate mobile and web apps are now using GNMT for 100% of machine translations from Chinese to English—about 18 million translations per day. The production deployment of GNMT was made possible by use of our publicly available machine learning toolkit TensorFlow and our Tensor Processing Units (TPUs), which provide sufficient computational power to deploy these powerful GNMT models while meeting the stringent latency requirements of the Google Translate product. Translating from Chinese to English is one of the more than 10,000 language pairs supported by Google Translate, and we will be working to roll out GNMT to many more of these over the coming months.

Machine translation is by no means solved. GNMT can still make significant errors that a human translator would never make, like dropping words and mistranslating proper names or rare terms, and translating sentences in isolation rather than considering the context of the paragraph or page. There is still a lot of work we can do to serve our users better. However, GNMT represents a significant milestone. We would like to celebrate it with the many researchers and engineers—both within Google and the wider community—who have contributed to this direction of research in the past few years.

Acknowledgements:
We thank members of the Google Brain team and the Google Translate team for the help with the project. We thank Nikhil Thorat and the Big Picture team for the visualization.

References:
[1] Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, Jeffrey Dean. Technical Report, 2016.
[2] Sequence to Sequence Learning with Neural Networks, Ilya Sutskever, Oriol Vinyals, Quoc V. Le. Advances in Neural Information Processing Systems, 2014.
[3] Addressing the rare word problem in neural machine translation, Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. Proceedings of the 53th Annual Meeting of the Association for Computational Linguistics, 2015.
[4] Neural Machine Translation by Jointly Learning to Align and Translate, Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio. International Conference on Learning Representations, 2015.
[5] Japanese and Korean voice search, Mike Schuster, and Kaisuke Nakajima. IEEE International Conference on Acoustics, Speech and Signal Processing, 2012.
[6] Neural Machine Translation of Rare Words with Subword Units, Rico Sennrich, Barry Haddow, Alexandra Birch. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016.

Of the 500 million+ people who use Google Translate, more than 9 in 10 live outside the U.S. We’ve talked with thousands of you in India, Indonesia, Brazil, and Thailand to learn what works and what doesn’t—and today we’re rolling out some big improvements.

 First, say hello to Tap to Translate on Android. We know millions of you painstakingly copy-paste text between Google Translate and other apps. Now, you can just copy the text of a chat, comment, song lyric, etc. in whichever app you’re using, and a translation will pop up right there—no need to switch apps:

Watch the video to learn more. Tap to Translate works for all 103 of Google Translate’s languages on any Android phone running Jellybean (4.2) and above.

Next, Offline Mode now works on iOS, and joins Android in using small offline packages. We know that many of you found the previous packages too big to download on unreliable data connections or to keep on your phone’s limited storage. That’s why we shrunk them by 90 percent, to a much more manageable 25 MB each.

Offline Mode is easy to set up: Just tap the arrow next to the language name to download the package for that language, and then you’ll be ready to do text translations whether you’re online or not—and it works with Tap to Translate too. We’ve just added a Filipino language pack, bringing our total number of offline languages to 52.

Finally, we’re adding Word Lens in Chinese. It’s our 29th language for instant visual translation, and it reads both to and from English, for both Simplified and Traditional Chinese. Try it on menus, signs, packages, and other printed text. As with all Word Lens languages, it works offline.

With Tap to Translate, improved Offline Mode, and Word Lens in Chinese, we hope you’ll find the latest version of Google Translate a helpful companion. These updates are rolling out over the next few days.

Posted by Barak Turovsky, Product Lead, Google Translate

Ten years ago, we launched Google Translate. Our goal was to break language barriers and to make the world more accessible. Since then we’ve grown from supporting two languages to 103, and from hundreds of users to hundreds of millions. And just like anyone’s first 10 years, we’ve learned to see and understand, talk, listen, have a conversation, write, and lean on friends for help.

But what we’re most inspired by is how Google Translate connects people in communities around the world, in ways we never could have imagined—like two farmers with a shared passion for tomato farming, a couple discovering they’re pregnant in a foreign country, and a young immigrant on his way to soccer stardom.

Here’s a look at Google Translate today, 10 years in:

1. Google Translate helps people make connections.
Translate can help people help each other, often in the most difficult of times. Recently we visited a community in Canada that is using Translate to break down barriers and make a refugee family feel more welcome:

2. There are more than 500 million of you using Google Translate.
The most common translations are between English and Spanish, Arabic, Russian, Portuguese and Indonesian.

3. Together we translate more than 100 billion words a day.

4. Translations reflect trends and events.
In addition to common phrases like “I love you,” we also see people looking for translations related to current events and trends. For instance, last year we saw a big spike in translations for the word “selfie,” and this past week, translations for “purple rain” spiked by more than 25,000 percent.

 5. You’re helping to make Google Translate better with Translate Community.
So far, 3.5 million people have made 90 million contributions through Translate Community, helping us improve and add new languages to Google Translate. A few properly translated sentences can make a huge difference when faced with a foreign language or country. By reviewing, validating and recommending translations, we’re able to improve the Google Translate on a daily basis.

6. Brazil uses Google Translate more than any other country.
Ninety-two percent of our translations come from outside of the United States, with Brazil topping the list.

7. You can see the world in your language.
Word Lens is your friend when reading menus, street signs and more. This feature in the Google Translate App lets you instantly see translations in 28 languages.

8. You can have a conversation no matter what language you speak.
In 2011, we first introduced the ability to have a bilingual conversation on Google Translate. The app will recognize which language is being spoken when you’re talking with someone, allowing you to have a natural conversation in 32 languages.

9. You don’t need an Internet connection to connect.
Many countries don’t have reliable Internet, so it’s important to be able to translate on the go. You can instantly translate signs and menus offline with Word Lens on both Android and iOS, and translate typed text offline with Android.

10. There’s always more to translate.
We’re excited and proud of what we’ve accomplished together over the last 10 years—but there’s lots more to do to break language barriers and help people communicate no matter where they’re from or what language they speak. Thank you for using Google Translate—here’s to another 10!

Posted by Barak Turovsky, Product Lead, Google Translate

Last week, we introduced 13 new languages to Google Translate. As we mentioned, there are a number of factors that go into adding a new language: once it’s established that it’s a written language with a significant amount of translations available on the web, we use a combination of machine learning, licensed content and input from Translate Community. One language where Translate Community played an especially big role was Frisian.

Today, we’re speaking with Sietske Poepjes, a member of the Frisian community who recently helped to organize a Translate Community event. Thanks to Sietske and the community’s support, we were able to get enough data to officially add Frisian to Google Translate. The interview gives an overview of what went into Sietske’s community effort and how you can get involved.

What’s your job and title? I’m a provincial deputy on language and education in the province of Fryslân. Fryslân is one of the twelve provinces in The Netherlands. It’s the only province with an official language other than Dutch, namely: Frisian.

Sietske Poepjes

Why is it important to add the Frisian language to Google Translate? Frisian is the second official language in the Netherlands and is the mother language of more than half of the population of the province of Fryslân. Most people in Fryslân are able to understand or speak the language, but not a lot of people are able to write it at a high level (only 15%). With most of our education taking place in Dutch, Google Translate can be used as a tool for those who can’t write the Frisian language sufficiently.

Besides, it’s important for lesser used languages like Frisian to be used digitally. Being one of the languages in Google Translate also enhances the visibility of the language and allows people throughout the world to translate to and from the language.

So you organized Frisian Google Translate Week last year — what motivated you to organize the event?To add Frisian to Google Translate, we knew we needed a lot of data. Since there wasn’t sufficient material available both in the Frisian and English language, Translate Community could help. To get everyone involved, the province of Fryslân decided to organise a central week in which everyone is asked to participate. The idea was that working together in the same week, we would motivate people to contribute even more. This has definitely paid out; thousands of people participated, resulting in nearly a million translated words!

Translate Community event.

How did you set up the event? Who did you work with?As the province of Fryslân, we coordinated the event with educational and scientific organisations and libraries, and received lots of support. The organisations invited employees, members, students or other interested people to come along and translate with them on a certain day/time. This turned out to work really well. One example: the Frisian department of the University of Groningen (which is outside of Fryslân) organised a reunion with former students and teachers and together they translated thousands of words.

We also organised an opening session which was the official start of the entire week and invited school children to help open the festival with their own song. At this opening session an introduction to Google Translate presentation was given by a Google Netherlands representative.

We provided hand-outs and made a training video on YouTube to guide people on how to navigate on Translate Community site and make contributions. We also created a commercial video (with famous Frisians), broadcast on the regional television “Omrop Fryslân”, which turned out to be very influential.

How did you motivate the participants?We relied on social media. We created our own Facebook event for people to join and asked all of the organisations involved to use their own social media to share our messages and calls for everyone to contribute. Participants could make a screenshot showing their number of contributions made, to share on Facebook. We gave the participant with the most contributions a Google Translate cake.

How did the Frisian week go? Any memorable moments?
The Frisian Google Translate Week became a huge success. A lot of Frisians participated, resulting in a whopping one million translated words at the end of the week. The total number of translations was revealed at a national festival for languages. It was amazing to see the amount of publicity we gained and to see that so many people were interested in our event. It was even broadcast in the Dutch news at prime time.

What’s the impact of the event? What are people’s reactions?
We saw the need for Google Translate, as we received a lot of feedback and questions from people who wanted to know when Frisian would be available in Google Translate.

The most impressive thing of the whole Frisian Google Translate Week is the commitment of all Frisians (in and outside the province of Fryslân). So many people participated and everyone felt the need to join and start translating. The Frisian community worked together to achieve a goal.

Have you worked on any follow up efforts?
Yes! We have organized a validation session. In this session, we have reached Frisian experts and gathered in the local provincial library to work together on validating the translations. It worked out really well, again the sense of community was very strong. And to thank everyone who participated in the Frisian Google Translate Week and the validation session, we organized a celebration party. It was a really nice party with a spectacular multilingual musical performance from the Frisian band ‘De Kast’. Their number-1 hit “De nije dei” (The new day) was performed with the lyrics in Dutch translated by Google Translate in the background. There were also secondary school pupils showing the use and work of Frisian in Google Translate to all guests.

The band De Kast, who performed songs in Frisian, Dutch and English with the meaning shown in other languages on the screen through Translate.

Any advice for future event organizers? 
We think it is very important to communicate about the value of Google Translate for the language, it improves the visibility of the language, and it offers speakers of the language a very helpful and easy-to-use digital tool. It shows the vitality of the language, which is especially important for small languages such as Frisian. Second, we think it is important to cooperate with the local organisations for a sense of community. And last but not least: we have made a nice event of it. It was great fun!

If you would like to help improve Google Translate and organize Translate Community events for your language, apply here.

Posted by Mengmeng Niu, Program Manager, Google Translate

In 2006, we started with machine learning-based translations between English and Arabic, Chinese and Russian. Almost 10 years later, with today’s update, we now offer 103 languages that cover 99% of the online population.

The 13 new languages — Amharic, Corsican, Frisian, Kyrgyz, Hawaiian, Kurdish (Kurmanji), Luxembourgish, Samoan, Scots Gaelic, Shona, Sindhi, Pashto and Xhosa — help bring a combined 120 million new people to the billions who can already communicate with Translate all over the world.

So what goes into adding a new language? Beyond the basic criteria that it must be a written language, we also need a significant amount of translations in the new language to be available on the web. From there, we use a combination of machine learning, licensed content and Translate Community.

As we scan the Web for billions of already translated texts, we use machine learning to identify statistical patterns at enormous scale, so our machines can “learn” the language. But, as already existing documents can’t cover the breadth of a language, we also rely on people like you in Translate Community to help improve current Google Translate languages and add new ones, like Frisian and Kyrgyz. So far, over 3 million people have contributed approximately 200 million translated words.

Before you dive into translating, here are a few fun facts about the new languages:

• Amharic (Ethiopia) is the second most widely spoken Semitic language after Arabic 
• Corsican (Island of Corsica, France) is closely related to Italian and was Napoleon’s first language 
• Frisian (Netherlands and Germany) is the native language of over half the inhabitants of the Friesland province of the Netherlands 
• Kyrgyz (Kyrgyzstan) is the language of the Epic of Manas, which is 20x longer than the Iliad and the Odyssey put together 
• Hawaiian (Hawaii) has lent several words to the English language, such as ukulele and wiki
• Kurdish (Kurmanji) (Turkey, Iraq, Iran and Syria) is written with Latin letters while the others two varieties of Kurdish are written with Arabic script 
• Luxembourgish (Luxembourg) completes the list of official EU languages Translate covers
• Samoan (Samoa and American Samoa) is written using only 14 letters 
• Scots Gaelic (Scottish highlands, UK) was introduced by Irish settlers in the 4th century AD
• Shona (Zimbabwe) is the most widely spoken of the hundreds of languages in the Bantu family
• Sindhi (Pakistan and India) was the native language of Muhammad Ali Jinnah, the “Father of the Nation” of Pakistan 
• Pashto (Afghanistan and Pakistan) is written in Perso-Arabic script with an additional 12 letters, for a total of 44 
• Xhosa (South Africa) is the second most common native language in the country after Afrikaans and features three kinds of clicks, represented by the letters x, q and c

We’ve come a long way with over 100 languages, but we aren’t done yet. If you want to help, International Mother Language day — just around the corner on February 21 — is a great time to get involved in Translate Community. To start, just select the languages you speak; then choose to either translate phrases on your own or validate existing translations. Every contribution helps improve the quality of translation over time. You can also share feedback directly from Translate.Google.com, so as you try out the new languages, we’d love to hear your suggestions.

For each new language, we make our translations better over time, both by improving our algorithms and systems and by learning from your translations with Translate Community. Today’s update will be rolling out over the coming days.

No matter what language you speak, we hope today’s update makes it easier to communicate with millions of new friends and break language barriers one conversation at a time.

Posted by Sveta Kelman, Senior Program Manager, Google Translate

Over the past year, more than one million people speaking 117 languages have made 50 million contributions through the Google Translate Community.

With those contributions we’ve launched 10 new languages, including Chichewa (Chinyanja) and Malayalam (മലയാളം), and been able to make improvements in how we speak dozens of other languages. Now almost 50% of the most common phrases typed in Google Translate come from translations provided by the Translate Community.

Translate Community members come from all over the world and translate in many different ways – from translating on their own to hosting group events. This year, Bengali speakers worked together to host events throughout the country by partnering with schools and cultural groups. And Frisian speakers worked with their government to create a week of events dedicated to getting their language added to Google Translate.

This month, language lovers are participating in a Translatathon in India. With just a few more days to go, if you speak Hindi, Bengali, Telugu, Marathi, Tamil, Gujarati, Kannada, Malayalam or Punjabi, sign up today to help Google Translate deliver better translations in your community.

Posted by Mengmeng Niu, Translate Community Program Manager

Sometimes language isn’t straightforward. Only a Hindi speaker could tell you that although ऊँट के मुँह में जीरा may literally mean ‘cumin seed in a camel’s mouth’, it actually means ‘a drop in the ocean’ or something too insufficient to fulfill a need.

There are 22 official languages in India. And while Google Translate can help you with nine of them at the moment, languages that are under-represented on the Internet like Bengali, Telugu, and Tamil could use a little help. This is where people who are passionate about their native languages can use the Translate Community tool to make a big difference.

We’ve just kicked off our second translatathon in India, this time for nine languages — Hindi, Bengali, Telugu, Marathi, Tamil, Gujarati, Kannada, Malayalam and Punjabi. You can use Google Translate Community on your phone, laptop or computer. Just type, swipe or tap translations in the languages you speak. You have the option to either translate phrases directly, or validate existing translations.

Last year 20,000 people contributed over one million new Hindi translations, helping improve the overall quality of Hindi content online. We’re now including all the Indic languages that Google Translate is available in, and we look forward to seeing how people from around the world can help Google say जंगल में मोर नाचा किस ने देखा? or আপনার পায়ে কুড়ল মারা more accurately. Millions of people in India are coming online for the first time and most of them don’t speak English. Bringing more Indian language content online, and improving Indian language translation quality, will help them have a better experience on the Web. 

Validate phrases with the Google Translate Community tool

Once you join the translatathon, you can Translate and validate words and short phrases up until December 30. We will then reward the 50 most active and accurate contributors with an Android One phone*.

So why not stop by and say नमस्ते, নমস্কার, வணக்கம் and help India showcase the beauty and diversity of languages online. Register and participate at g.co/translatathon and thanks in advance for your help. You’re making the web better for everyone.

*Terms and conditions apply: https://goo.gl/kEAehI

Posted by Barak Turovsky, Product Lead, Google Translate

We’re always amazed by the power of technology to connect people. Not long ago we heard a story involving the Google Translate app and a boy named Alberto who had recently moved from Spain to a small town in Northern Ireland, with little knowledge of English. When Alberto joined Portadown’s youth soccer club, his coaches Gary and Glen turned to Google Translate to communicate with Alberto and his mother, on and off the field. As they progressed from protección de la pelota to retroceso de bicicleta, Alberto grew to feel a part of the team. We loved this story (and wanted to share it with you) because what Gary and Glen did was so much bigger than translating sentences from one language into another. They didn’t just find a way to coach Alberto in football—they found a way to invite someone who was on the outside into their community.

¡Vamos, Alberto!

Posted by Barak Turovsky, Product Lead, Google Translate

We’re all about breaking language barriers, whatever language you speak or device you use. So with that in mind, over the next week, we’ll be rolling out two new Google Translate app features— instantly translating both English and German to Arabic and easier multitasking for iPad users.

See the world in a new language with instant visual translation 
You can already have bilingual conversations from English or German to Arabic thanks to the conversation mode or text input in the Google Translate app. Today, we’re also adding the ability to translate printed text instantly between these languages.

To use instant visual translation, just open the app, click on the camera, and point it at the text you need to translate. You’ll see the text transform from one language to another in real-time on your screen. And the best part? There’s no Internet connection or cell phone data needed.

To try out Arabic with either English or German you’ll be prompted to download a small (~2 MB) language pack.

Split View translations with the newest iPads
Starting today, customers using iPads supporting Split View will be able to use Google Translate along with the new feature. So if you’re sending an email or text and need to translate, you can see both apps at the same time. And it even works with text from online books or websites.

Posted by Barak Turovsky, Product Lead, Google Translate

We face communication barriers every day. Switching back and forth between apps and screens to translate shouldn’t be another one. We’ve heard your feedback, and have worked with the Android team to make translating text, chats, and other app content a whole lot easier.

Beginning this week, you’ll be able to translate in 90 languages right from within some of your favorite apps like TripAdvisor, WhatsApp and LinkedIn.

This update works on any device running the newest version of Android’s operating system (Android 6.0, Marshmallow). To get started, you first need to have the Translate app downloaded on your Android phone. From there, just go to an app, like TripAdvisor or LinkedIn, and highlight and select the text you want to translate. This feature is already enabled in apps that use Android text selection behavior. Developers who created custom text selection behavior can also easily add the new feature.

More than 500 million people translate over 100 billion words a day on Google Translate. With updates like this one, plus features like conversation mode and instant camera translation, we’re making Translate available anywhere you need it. So when you’re chatting with a new colleague from halfway around the world, conversation mode is perfect. Wondering which subway sign says “exit” on your next global adventure? Instant camera translation has your back. And now, when you’re sending messages or checking out reviews on your phone, you can translate right from within the apps you’re using.

Posted by Barak Turovsky, Product Lead, Google Translate

More than 500 million people use Google Translate every month across web and mobile phones, translating more than 100 billion words every day around the globe. Now, we’re launching Google Translate on all Android Wear watches, too.

Translate is built into the latest Android Wear software update, so you can have bilingual conversations even if you don’t have Google Translate on your phone, or if you’re away from your phone but connected via Wi-Fi.

And it’s easy to use – just speak into your watch to see your conversation translated into any of 44 languages. Flip your wrist to show the translation to a friend. When they respond in their own language, flip your wrist back, and you’ll see in your language what they’ve just said. Google Translate will automatically recognize which of the two languages is being spoken, so once you tap to start the conversation, all you and your buddy need to do is keep talking naturally.

Google Translate covers 90 languages total (for text translation), and we are always working to expand the number of languages that work across various features.

Posted by Barak Turovsky, Product Lead, Google Translate and Nathan Beach, Product Manager, Android Wear

Today we announced that the Google Translate app now does real-time visual translation of 20 more languages. So the next time you’re in Prague and can’t read a menu, we’ve got your back. But how are we able to recognize these new languages?

In short: deep neural nets. When the Word Lens team joined Google, we were excited for the opportunity to work with some of the leading researchers in deep learning. Neural nets have gotten a lot of attention in the last few years because they’ve set all kinds of records in image recognition. Five years ago, if you gave a computer an image of a cat or a dog, it had trouble telling which was which. Thanks to convolutional neural networks, not only can computers tell the difference between cats and dogs, they can even recognize different breeds of dogs. Yes, they’re good for more than just trippy art—if you’re translating a foreign menu or sign with the latest version of Google’s Translate app, you’re now using a deep neural net. And the amazing part is it can all work on your phone, without an Internet connection. Here’s how.

Step by step
First, when a camera image comes in, the Google Translate app has to find the letters in the picture. It needs to weed out background objects like trees or cars, and pick up on the words we want translated. It looks at blobs of pixels that have similar color to each other that are also near other similar blobs of pixels. Those are possibly letters, and if they’re near each other, that makes a continuous line we should read.

Second, Translate has to recognize what each letter actually is. This is where deep learning comes in. We use a convolutional neural network, training it on letters and non-letters so it can learn what different letters look like.

But interestingly, if we train just on very “clean”-looking letters, we risk not understanding what real-life letters look like. Letters out in the real world are marred by reflections, dirt, smudges, and all kinds of weirdness. So we built our letter generator to create all kinds of fake “dirt” to convincingly mimic the noisiness of the real world—fake reflections, fake smudges, fake weirdness all around.

Why not just train on real-life photos of letters? Well, it’s tough to find enough examples in all the languages we need, and it’s harder to maintain the fine control over what examples we use when we’re aiming to train a really efficient, compact neural network. So it’s more effective to simulate the dirt.

Some of the “dirty” letters we use for training. Dirt, highlights, and rotation, but not too much because we don’t want to confuse our neural net.

The third step is to take those recognized letters, and look them up in a dictionary to get translations. Since every previous step could have failed in some way, the dictionary lookup needs to be approximate. That way, if we read an ‘S’ as a ‘5’, we’ll still be able to find the word ‘5uper’.

Finally, we render the translation on top of the original words in the same style as the original. We can do this because we’ve already found and read the letters in the image, so we know exactly where they are. We can look at the colors surrounding the letters and use that to erase the original letters. And then we can draw the translation on top using the original foreground color.

Crunching it down for mobile
Now, if we could do this visual translation in our data centers, it wouldn’t be too hard. But a lot of our users, especially those getting online for the very first time, have slow or intermittent network connections and smartphones starved for computing power. These low-end phones can be about 50 times slower than a good laptop—and a good laptop is already much slower than the data centers that typically run our image recognition systems. So how do we get visual translation on these phones, with no connection to the cloud, translating in real-time as the camera moves around?

We needed to develop a very small neural net, and put severe limits on how much we tried to teach it—in essence, put an upper bound on the density of information it handles. The challenge here was in creating the most effective training data. Since we’re generating our own training data, we put a lot of effort into including just the right data and nothing more. For instance, we want to be able to recognize a letter with a small amount of rotation, but not too much. If we overdo the rotation, the neural network will use too much of its information density on unimportant things. So we put effort into making tools that would give us a fast iteration time and good visualizations. Inside of a few minutes, we can change the algorithms for generating training data, generate it, retrain, and visualize. From there we can look at what kind of letters are failing and why. At one point, we were warping our training data too much, and ‘$’ started to be recognized as ‘S’. We were able to quickly identify that and adjust the warping parameters to fix the problem. It was like trying to paint a picture of letters that you’d see in real life with all their imperfections painted just perfectly.

To achieve real-time, we also heavily optimized and hand-tuned the math operations. That meant using the mobile processor’s SIMD instructions and tuning things like matrix multiplies to fit processing into all levels of cache memory.

In the end, we were able to get our networks to give us significantly better results while running about as fast as our old system—great for translating what you see around you on the fly. Sometimes new technology can seem very abstract, and it’s not always obvious what the applications for things like convolutional neural nets could be. We think breaking down language barriers is one great use.

Posted by Otavio Good, Software Engineer, Google Translate

(Cross-posted on the Google Research Blog)

The Google Translate app already lets you instantly visually translate printed text in seven languages. Just open the app, click on the camera, and point it at the text you need to translate—a street sign, ingredient list, instruction manual, dials on a washing machine. You’ll see the text transform live on your screen into the other language. No Internet connection or cell phone data needed.

Today, we’re updating the Google Translate app again—expanding instant visual translation to 20 more languages (for a total of 27!), and making real-time voice translations a lot faster and smoother—so even more people can experience the world in their language.

Instantly translate printed text in 27 languages
We started out with seven languages—English, French, German, Italian, Portuguese, Russian and Spanish—and today we’re adding 20 more. You can now translate to and from English and Bulgarian, Catalan, Croatian, Czech, Danish, Dutch, Filipino, Finnish, Hungarian, Indonesian, Lithuanian, Norwegian, Polish, Romanian, Slovak, Swedish, Turkish and Ukrainian. You can also do one-way translations from English to Hindi and Thai. (Or, try snapping a pic of the text you’d like translated—we have a total of 37 languages in camera mode.)

To try out the new languages, go to the Google Translate app, set “English” along with the language you’d like to translate, and click the camera button; you’ll be prompted to download a small (~2 MB) language pack for each.

Ready to see all of these languages in action?

And how exactly did we get so many new languages running on a device with no data connection? It’s all about convolutional neural networks (whew)—geek out on that over on our Research blog.

Have a natural, smoother conversation—even with a slower mobile network
In many emerging markets, slow mobile networks can make it challenging to access many online tools – so if you live in an area with unreliable mobile networks, our other update today is for you. In addition to instant visual translation, we’ve also improved our voice conversation mode (enabling real-time translation of conversations across 32 languages), so it’s even faster and more natural on slow networks.

These updates are coming to both Android and iOS, rolling out over the next few days.

Translate Community helps us get better every day
On top of today’s updates, we’re also continuously working to improve the quality of the translations themselves and to add new languages. A year ago this week, we launched Translate Community, a place for multilingual people from anywhere in the world to provide and correct translations. Thanks to the millions of language lovers who have already pitched in—more than 100 million words so far!—we’ve been updating our translations for over 90 language pairs, and plan to update many more as our community grows.

We’ve still got lots of work to do: more than half of the content on the Internet is in English, but only around 20% of the world’s population speaks English. Today’s updates knock down a few more language barriers, helping you communicate better and get the information you need.

Posted by Barak Turovsky, Product Lead, Google Translate

(Cross-posted on the Official Google Blog)

Posted by Otavio Good, Software Engineer, Google Translate

Today we announced that the Google Translate app now does real-time visual translation of 20 more languages. So the next time you’re in Prague and can’t read a menu, we’ve got your back. But how are we able to recognize these new languages?

In short: deep neural nets. When the Word Lens team joined Google, we were excited for the opportunity to work with some of the leading researchers in deep learning. Neural nets have gotten a lot of attention in the last few years because they’ve set all kinds of records in image recognition. Five years ago, if you gave a computer an image of a cat or a dog, it had trouble telling which was which. Thanks to convolutional neural networks, not only can computers tell the difference between cats and dogs, they can even recognize different breeds of dogs. Yes, they’re good for more than just trippy art—if you’re translating a foreign menu or sign with the latest version of Google’s Translate app, you’re now using a deep neural net. And the amazing part is it can all work on your phone, without an Internet connection. Here’s how.

Step by step

First, when a camera image comes in, the Google Translate app has to find the letters in the picture. It needs to weed out background objects like trees or cars, and pick up on the words we want translated. It looks at blobs of pixels that have similar color to each other that are also near other similar blobs of pixels. Those are possibly letters, and if they’re near each other, that makes a continuous line we should read.

Second, Translate has to recognize what each letter actually is. This is where deep learning comes in. We use a convolutional neural network, training it on letters and non-letters so it can learn what different letters look like.

But interestingly, if we train just on very “clean”-looking letters, we risk not understanding what real-life letters look like. Letters out in the real world are marred by reflections, dirt, smudges, and all kinds of weirdness. So we built our letter generator to create all kinds of fake “dirt” to convincingly mimic the noisiness of the real world—fake reflections, fake smudges, fake weirdness all around.

Why not just train on real-life photos of letters? Well, it’s tough to find enough examples in all the languages we need, and it’s harder to maintain the fine control over what examples we use when we’re aiming to train a really efficient, compact neural network. So it’s more effective to simulate the dirt.

Some of the “dirty” letters we use for training. Dirt, highlights, and rotation, but not too much because we don’t want to confuse our neural net.

The third step is to take those recognized letters, and look them up in a dictionary to get translations. Since every previous step could have failed in some way, the dictionary lookup needs to be approximate. That way, if we read an ‘S’ as a ‘5’, we’ll still be able to find the word ‘5uper’.

Finally, we render the translation on top of the original words in the same style as the original. We can do this because we’ve already found and read the letters in the image, so we know exactly where they are. We can look at the colors surrounding the letters and use that to erase the original letters. And then we can draw the translation on top using the original foreground color.

Crunching it down for mobile

Now, if we could do this visual translation in our data centers, it wouldn’t be too hard. But a lot of our users, especially those getting online for the very first time, have slow or intermittent network connections and smartphones starved for computing power. These low-end phones can be about 50 times slower than a good laptop—and a good laptop is already much slower than the data centers that typically run our image recognition systems. So how do we get visual translation on these phones, with no connection to the cloud, translating in real-time as the camera moves around?

We needed to develop a very small neural net, and put severe limits on how much we tried to teach it—in essence, put an upper bound on the density of information it handles. The challenge here was in creating the most effective training data. Since we’re generating our own training data, we put a lot of effort into including just the right data and nothing more. For instance, we want to be able to recognize a letter with a small amount of rotation, but not too much. If we overdo the rotation, the neural network will use too much of its information density on unimportant things. So we put effort into making tools that would give us a fast iteration time and good visualizations. Inside of a few minutes, we can change the algorithms for generating training data, generate it, retrain, and visualize. From there we can look at what kind of letters are failing and why. At one point, we were warping our training data too much, and ‘$’ started to be recognized as ‘S’. We were able to quickly identify that and adjust the warping parameters to fix the problem. It was like trying to paint a picture of letters that you’d see in real life with all their imperfections painted just perfectly.

To achieve real-time, we also heavily optimized and hand-tuned the math operations. That meant using the mobile processor’s SIMD instructions and tuning things like matrix multiplies to fit processing into all levels of cache memory.

In the end, we were able to get our networks to give us significantly better results while running about as fast as our old system—great for translating what you see around you on the fly. Sometimes new technology can seem very abstract, and it’s not always obvious what the applications for things like convolutional neural nets could be. We think breaking down language barriers is one great use.

People use Google Translate a whole lot—we translate over 100 billion words a day! However, in the past, our translation systems have generally been better at making sense of government and business documents than in helping people casually communicate.

But that’s all changing thanks to people like you and a recent update we rolled out. So the next time you translate informal speech in Google Translate, you might just find a better translation. Here’s an example of how it’s improved:

So how exactly are people like you impacting Google Translate? Well, with Translate Community hundreds of thousands of people have generously donated their time in service of cross-language communication. It’s fun and really easy: tell us what languages you speak; choose to either see a phrase and translate it on your own or correct current translations already in the system. Based on translations from the community, we will incorporate corrections and over time learn the language a little better.

There’s a whole lot more work to do, but with more help from everyday people through Translate Community, we can continue to improve the 90 languages we already speak and keep adding more.

Posted by Aaron Babst, Community Program Manager, Google Translate