Google Data » Carol Smith https://googledata.org Everything Google: News, Products, Services, Content, Culture Fri, 10 Apr 2015 22:28:52 +0000 en-US hourly 1 http://wordpress.org/?v=4.1.1 Google Code-in Wrapup https://googledata.org/google-open-source/google-code-in-wrapup/ https://googledata.org/google-open-source/google-code-in-wrapup/#comments Tue, 11 Jan 2011 21:08:00 +0000
This was a great year for Google Code-In. We had a total of 361 students complete at least one task during our contest period. We are still compiling the statistics on our participants and plan to post a follow-up once we are done reviewing tasks soon. We are quite proud of the participation in the contest this year and hope many of our student participants will go on to continue contributing to the organizations for which they completed tasks.

Grand prize winners will be announced on February 14. Stay tuned to this blog for the announcement!

Thank you to all our organization administrations, mentors, and students for your participation this year!

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/google-code-in-wrapup/feed/ 0 Google Code-in Check-in https://googledata.org/google-open-source/google-code-in-check-in/ https://googledata.org/google-open-source/google-code-in-check-in/#comments Mon, 13 Dec 2010 18:49:00 +0000
We wanted to do a quick check-in on Google Code-in to let you know how the contest is going. We are just over four weeks into the contest – more than halfway. We’re quite excited about the participation thus far and hope more of you are planning to get involved in the coming days.

As of today we have more than 290 participants who have completed at least one task. There have been 813 tasks completed by all our student participants so far for a combined point total of 1,605. Points are calculated according to difficulty of the task: “Hard” tasks are worth 4 points, “Medium” tasks are worth 2 points, and “Easy” tasks are worth 1 point.

We’re also quite impressed with the international representation we’ve gotten from the contest – over 75% of our participants are from outside the United States. Our top 10 participating countries in order are: United States, Romania, Bulgaria, Russian Federation, Poland, Canada, Germany, India, Italy, and Belarus.

We would love to have more students participate! There are currently over 400 tasks that are unclaimed and need someone to work on them.

Remember, the contest ends on January 10, 2011. Don’t delay, claim a task today!

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/google-code-in-check-in/feed/ 0 Google Code-in: Contest is Open! https://googledata.org/google-open-source/google-code-in-contest-is-open/ https://googledata.org/google-open-source/google-code-in-contest-is-open/#comments Mon, 22 Nov 2010 18:45:00 +0000

We’re pleased to announce that today marks the start of the Google Code-in contest. Students who are at least 13 years old and not more than 18 years old and are currently enrolled in a pre-university school on November 22, 2010 are eligible to participate. Start claiming tasks to work on for our participating mentor organizations right now.

Tasks are sorted according to organization, difficulty level, type of task, and point value. If you are considering participating, take a look at the tasks list and see what you’re interested in claiming. You can comment on the task if you have questions about the task for the organization.

For successful completion of just one task you will receive a t-shirt and certificate of participation. For every three tasks you complete you’ll receive $100, up to a maximum of $500. Our 10 contestants with the most points at the end of the contest win the Grand Prize: an all-expenses paid trip with a family member to Google’s Headquarters in Mountain View, California!

The contest ends on January 10, 2011. So don’t delay, claim a task today!

If you have questions, please review our Frequently Asked Questions and the Rules page, or take a look at the video above. You can also join our discussion list and ask your question there.

Have fun, and meet you on the list or in Mountain View!

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/google-code-in-contest-is-open/feed/ 0 Announcing Accepted Organizations for Google Code-in https://googledata.org/google-open-source/announcing-accepted-organizations-for-google-code-in/ https://googledata.org/google-open-source/announcing-accepted-organizations-for-google-code-in/#comments Fri, 05 Nov 2010 16:05:00 +0000

We’re pleased to announce that today we have chosen 20 open source organizations to participate as mentoring organizations in Google Code-in.

Here’s a list of our participating organizations this year:

1. The Apertium Project
2. The Battle for Wesnoth
3. Debian Project
4. Dragonfly BSD
5. Drupal
6. GNOME
7. Haiku
8. KDE
9. LimeSurvey
10. MoinMoin
11. Mono Project
12. OSUOSL
13. Parrot Foundation and The Perl Foundation
14. Plone Foundation
15. RTEMS Project
16. Sahana Software Foundation
17. Tux4Kids
18. VideoLAN
19. WordPress
20. WorldForge

These organizations will be creating tasks for the student participants in the contest to work on and submit to earn prizes. We’ll be having tasks from all 8 categories in our task tracker: code, documentation, outreach, quality assurance, research, training, translation, user interface.

We encourage you to sign up now if you would like to participate in the contest! Simply visit Melange and click on “Sign In.” Log in or create a Google account, and then click on “Create Profile” on the left. Once you’ve done that you’re ready to start claiming tasks on November 22!

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/announcing-accepted-organizations-for-google-code-in/feed/ 0 Google Code-in: School’s Out, Code’s In! https://googledata.org/google-open-source/google-code-in-school%e2%80%99s-out-code%e2%80%99s-in/ https://googledata.org/google-open-source/google-code-in-school%e2%80%99s-out-code%e2%80%99s-in/#comments Thu, 07 Oct 2010 18:00:00 +0000

We are very pleased to announce Google Code-in, an open source development and outreach contest targeted at 13-18 year old students around the world.

Some of you may remember the pilot program from 2007-2008 we ran called the Google Highly Open Participation Contest. This contest gave 400 students around the world an opportunity to help out open source projects on the following kinds of tasks:

1. Code: Tasks related to writing or refactoring code
2. Documentation: Tasks related to creating/editing documents
3. Outreach: Tasks related to community management and outreach/marketing
4. Quality Assurance: Tasks related to testing and ensuring code is of high quality
5. Research: Tasks related to studying a problem and recommending solutions
6. Training: Tasks related to helping others learn more
7. Translation: Tasks related to localization
8. User Interface: Tasks related to user experience research or user interface design and interaction

It was a huge success, and we’re looking forward to another great year ahead of us with the contest with its new name, Google Code-in.

Be sure to check out our Frequently Asked Questions about the contest for answers to your questions about participating.We’re hoping to get pre-university students from all over the world involved. So please help us spread the word.

Stay tuned to this blog and to our mailing list for more updates on the contest. We will announce the mentoring organizations that are participating on November 5. The contest starts on November 22, 2010!

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/google-code-in-school%e2%80%99s-out-code%e2%80%99s-in/feed/ 0 Ohio LinuxFest Wrap-up https://googledata.org/google-open-source/ohio-linuxfest-wrap-up/ https://googledata.org/google-open-source/ohio-linuxfest-wrap-up/#comments Mon, 20 Sep 2010 23:27:00 +0000

I just got back from Ohio LinuxFest, a three-day all-volunteer conference on free and open source software held in Columbus, Ohio. A wonderful group of about 800 people, some from the Midwest, some of whom were from other parts of the country, some of whom were experienced with open source software, and some of whom were new to it, attended. I enjoyed seeing the wide variety of talk topics - from starting your own business in open source to an entire workshop devoted to diversity in open source software development. Some of the topics were even a little unexpected - there was a great talk on reaching out to new people through video games.

I spoke on the 7 Habits of Highly Ineffective Project Managers late in the day on Saturday. Much of the content for the talk was gleaned from experience at Google and managing the Google Summer of Code project. The audience had some great questions about how to effectively manage upward (to your boss), what sorts of tactics to take when you feel your managers are meddling too much in your project, and how to manage your workflow in an environment where too many tasks are high priority. I’ll be giving this talk again at linux.conf.au, which is being held in Brisbane January 24th to 29th next year.

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/ohio-linuxfest-wrap-up/feed/ 0 Our 6th Google Summer of Code Has Come to an End https://googledata.org/google-open-source/our-6th-google-summer-of-code-has-come-to-an-end/ https://googledata.org/google-open-source/our-6th-google-summer-of-code-has-come-to-an-end/#comments Tue, 24 Aug 2010 23:47:00 +0000

We’ve just finished our 6th year of Google Summer of Code™, our innovative program designed to introduce students at colleges and universities around the world to open source software development. Over 2000 mentors and over 1000 students from 69 countries began working together on over 150 open source software projects, and we're happy to announce that 89% of our student participants have received passing final evaluations, which is about 4% better than 2009. This is our best success rate to date.

These successful students are now preparing code samples to present to the rest of the world; we'll post an update here when the source code produced during this year's Google Summer of Code has been made available on project hosting on Google Code. Of course, there's no need to wait for code samples - you can check out their work by visiting the websites and mailing lists of the participating projects now. We'll also be publishing more extensive statistics from our program evaluations, along with wrap up reports from some of our participating mentoring organizations, so stay tuned for more details in the coming weeks.

Congratulations to all of our students for their hard work this summer. We hope you will continue working with your project communities with source code, documentation, and enthusiasm long after this summer has ended. Many thanks also to our community of mentors whose time, skill and dedication make this program possible.

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/our-6th-google-summer-of-code-has-come-to-an-end/feed/ 0 Google Summer of Code Midterm Evaluations https://googledata.org/google-open-source/google-summer-of-code-midterm-evaluations/ https://googledata.org/google-open-source/google-summer-of-code-midterm-evaluations/#comments Thu, 05 Aug 2010 17:21:00 +0000
Midterm evaluations for Google Summer of Code 2010™ have wrapped up and we have some great news about the program.

Out of the over 1,000 participating students from the beginning of the program, 964 have passed their midterm evaluation. That’s just a over 90% pass rate - exactly on target for what we expect from the program.

Since Google Summer of Code started in 2005, we’ve had over 5,000 students complete the Google Summer of Code program. Take a look at the timeline on our website for more details about the program - our final evaluations are approaching!

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/google-summer-of-code-midterm-evaluations/feed/ 0 Google Summer of Code BoF at USENIX https://googledata.org/google-open-source/google-summer-of-code-bof-at-usenix/ https://googledata.org/google-open-source/google-summer-of-code-bof-at-usenix/#comments Mon, 12 Jul 2010 15:00:00 +0000
USENIX is a technical organization that has a lot of community members associated with our open source efforts as well as lots of strong ties to the research community. A couple weeks ago I attended the USENIX Annual Technical Conference (ATC). Googler Dave Presotto was a member of the program committee. ATC is a federated conference that brings together researchers and developers working in a wide variety of focus areas. We love to support the members of this organization in their efforts throughout the year. Among the presentations I enjoyed most were the one on robotic honeybees, another on power usage in smart phones, and really enjoyed the keynote on concurrency from Ivan Sutherland.

We held a Google Summer of Code™ Birds of a Feather (BoF) meetup on Thursday night after the conference reception. We talked about Google Summer of Code over ice cream and beer well into the evening. In all we had about 25 attendees, many of whom hadn’t heard of Google Summer of Code before. It was great to tell some people new to the program about it and also to hear from those who have participated about how its changed their lives. A great time was had by all; we even ended up closing the place down that evening.

Don’t miss the next Google Summer of Code BoF at OSCON in just a couple weeks!

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/google-summer-of-code-bof-at-usenix/feed/ 0 Google Summer of Code 2010: Meet The Students and Mentors! https://googledata.org/google-open-source/google-summer-of-code-2010-meet-the-students-and-mentors/ https://googledata.org/google-open-source/google-summer-of-code-2010-meet-the-students-and-mentors/#comments Fri, 02 Jul 2010 16:39:00 +0000

Following up on my post from a few weeks ago, I’d like to give you all some more statistics about our Google Summer of Code™ program participants this year.

• We have 69 student countries represented this year. New countries represented by students include Jamaica, Morocco, and Cambodia.
• For the first time we have mentors from Chile, South Africa, Taiwan, and Peru.
• We have mentors from 52 different countries this year.
• We had 3,464 students submit a total of 5,539 proposals in all. Last year we had 5,885 proposals submitted by 3,496 students.
• The open source organizations participating this year received an average of 36 proposals to review. We have 150 participating organizations this year.

We accepted 1,026 of those proposals to become full Google Summer of Code participant projects this year. This is 26 more than we had planned for but there were so many great applications this time that we just couldn't stop at 1000. We have 943 mentors for the students’ projects this year, which means we don’t have quite a 1:1 ratio of students to mentors; some organizations choose to co-mentor students or have the whole organization mentor all the students who participate.

Midterm evaluations are almost upon us. Check out the timeline to see what’s coming up for Google Summer of Code!

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/google-summer-of-code-2010-meet-the-students-and-mentors/feed/ 0 Birds of a Feather at Open Source Bridge https://googledata.org/google-open-source/birds-of-a-feather-at-open-source-bridge/ https://googledata.org/google-open-source/birds-of-a-feather-at-open-source-bridge/#comments Thu, 17 Jun 2010 22:56:00 +0000
The second annual Open Source Bridge, held in Portland, Oregon, was a blast. I presented a talk on Foundations, Non-Profits, and Open Source and I participated in some great sessions as well on topics ranging from How to Give a Great Tech Talk (including the 7 Habits of Highly Ineffective Speakers) and a geek choir.

The best part of the conference in my opinion, though, was the Google Summer of Code™ Birds of a Feather (“BoF”) session that we held on Thursday night. I and my fellow Googlers, Ellen Ko and Cat Allman, met some past and present students and mentors and also talked with some enthusiastic students who may be applying next year. There was, of course, the traditional post-BoF gathering at Old Town Pizza, as well. We all had some great conversations, made some new friends and are looking forward to the Google Summer of Code BoF at OSCON on Wednesday July 21st.

Hope to see you there!

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/birds-of-a-feather-at-open-source-bridge/feed/ 0 Google Summer of Code 2010: Coding Starts Today! https://googledata.org/google-open-source/google-summer-of-code-2010-coding-starts-today/ https://googledata.org/google-open-source/google-summer-of-code-2010-coding-starts-today/#comments Mon, 24 May 2010 16:02:00 +0000

Starting today all our students will formally begin coding their projects for Google Summer of Code™. Our students’ projects this year range from compilers to mobile applications, from web crawlers to virtual clusters, from APIs to social networking improvements, and more. We're really excited about the ever increasing variety of cool projects happening this year.

Take a look at the timeline to find out more about what’s coming up for Google Summer of Code.

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/google-summer-of-code-2010-coding-starts-today/feed/ 0 The Ashpocalypse https://googledata.org/google-open-source/the-ashpocalypse/ https://googledata.org/google-open-source/the-ashpocalypse/#comments Mon, 17 May 2010 16:00:00 +0000 Melange is the open source web app that runs Google Summer of Code™ each year. It is being developed and maintained by a volunteer team of student developers in several different countries. I try to meet with the Melange developers about every quarter in person to get hands-on time with the tool, suggest improvements, and give them time to code together and work out problems in person. It is usually relatively simple to bring us all together for a productive week, and get everyone home again. But then, usually we don’t have an erupting volcano to schedule around.

My trip to the Netherlands started out simply enough: I flew to Amsterdam, and took a train to Delft. On Sunday evening, I met up with the Melange developers for dinner. We started work bright and early on Monday morning, and spent the next few days working on Google Summer of Code and the Melange tool.

Unfortunately, while we were at TU Delft on Thursday, reports of airport closures started to reach us. By Friday morning, we were assured by the respective airlines that none of our developers would be flying out of Amsterdam that evening.

We took the train back to Amsterdam and spent the night in a hotel. On Saturday morning we had breakfast and went over to Amsterdam Centraal in the hopes of getting train tickets back to our international destinations. We got there around 11am to find that the wait was already three hours long and the line snaked out the door and through the station:



We waited. And waited. And waited.



The moment our number was called, we all ran over to the counter. Mario took a picture. That's our number at the top (C037):



Sverre and Lennard, based in the Netherlands, were both lucky and able to travel back home by local train and bike. But eventually Mario, Madhusudan, and Daniel got their respective train tickets home as well.

After all the developers were safely en route, I started my own trip. I ended up opting for a ferry ride from mainland Europe to England, in the hopes of getting a flight home from there. My wait in Calais was just as bad as the wait in Amsterdam Centraal. Another 3-hour line, in Calais:



By the time I got to London, most airports had reopened. My flight home was uneventful. I promptly slept for 12 hours once happily ensconced in my own bed.

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/the-ashpocalypse/feed/ 0 Google Summer of Code 2010: Meet the Students! https://googledata.org/google-open-source/google-summer-of-code-2010-meet-the-students/ https://googledata.org/google-open-source/google-summer-of-code-2010-meet-the-students/#comments Wed, 12 May 2010 16:00:00 +0000 We’ve had a great showing from students around the world for Google Summer of Code™ this year. In all, we accepted 1,026 students from 69 countries worldwide. Our top ten countries by number of students accepted this year are: United States (197), India (125), Germany (57), Brazil (50), Poland (46), Canada (40), China (39), United Kingdom (36), France (35), Sri Lanka(34).


145 of the students this year also completed projects as part of Google Summer of Code in 2009.

Meanwhile, since acceptance, all the students have been getting to know their mentors and organizations. Our Community Bonding Period ends on May 24 and coding officially starts the same day. You can find more about the upcoming events in our timeline.

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/google-summer-of-code-2010-meet-the-students/feed/ 0 This Year’s Google Summer of Code Students Announced! https://googledata.org/google-open-source/this-year%e2%80%99s-google-summer-of-code-students-announced/ https://googledata.org/google-open-source/this-year%e2%80%99s-google-summer-of-code-students-announced/#comments Mon, 26 Apr 2010 16:30:00 +0000 Accepted students for Google Summer of Code™ have been announced! Congratulations to all the students accepted for this year’s program. We’re looking forward to a great summer.

If you would like more information on what’s coming up next for Google Summer of Code, check out our program timeline.

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/this-year%e2%80%99s-google-summer-of-code-students-announced/feed/ 0 Google Summer of Code 2010 Info Session at Universidade Estadual de Campinas –UNICAMP https://googledata.org/google-open-source/google-summer-of-code-2010-info-session-at-universidade-estadual-de-campinas-%e2%80%93unicamp/ https://googledata.org/google-open-source/google-summer-of-code-2010-info-session-at-universidade-estadual-de-campinas-%e2%80%93unicamp/#comments Mon, 19 Apr 2010 16:00:00 +0000

There were about 70 attendees and the session lasted an hour and a half. The first 20-30 minutes were spent on presenting the GPSL FOSS Seminars Series, since this was the first talk of the series for this semester, and going through the official Google Summer of Code™ program slides. The speakers spent the last hour in an open chat with the attendees about their personal experiences with the program.

There were 8 speakers in total:
Bruno Cardoso Lopes — LLVM student 2007/2008/2009;
Eduardo Lima — BlueZ mentor 2008 and Maemo mentor 2009;
Gustavo Padovan — BlueZ student 2009;
Helder Ribeiro — Ruby Central student 2007 and ReviewBoard student 2009;
João Batista Corrêa Gomes Moreira — OSSIM student 2008 and NMAP student 2009; João Paulo Rechi Vita — BlueZ student 2008/2009;
Leonardo Piga — Xiph.Org student 2008;
Luis Felipe Strano Moraes — BBC student 2006 and Maemo admin/mentor 2009.



By João Paulo Rechi Vita, UNICAMP
]]> https://googledata.org/google-open-source/google-summer-of-code-2010-info-session-at-universidade-estadual-de-campinas-%e2%80%93unicamp/feed/ 0 Google Summer of Code Student Applications Now Closed! https://googledata.org/google-open-source/google-summer-of-code-student-applications-now-closed/ https://googledata.org/google-open-source/google-summer-of-code-student-applications-now-closed/#comments Mon, 12 Apr 2010 16:00:00 +0000

Our application deadline for student applications to Google Summer of Code™ has now passed. Thank you to all the students who applied this year! We got 5,539 proposals in all.

If you are a student waiting to find out if you will be accepted into the program, be sure to check out the Google Summer of Code site on April 26 at 19:00 UTC for the announcements.

Our community bonding period starts right after announcements are made, and then, as mentioned in our timeline, coding officially starts on May 24. Good luck to all the applicants we got this year, and we’re looking forward to a great summer!

By Carol Smith, Open Source Programs Office
]]> https://googledata.org/google-open-source/google-summer-of-code-student-applications-now-closed/feed/ 0 Google Annotations Gallery https://googledata.org/google-open-source/google-annotations-gallery/ https://googledata.org/google-open-source/google-annotations-gallery/#comments Thu, 01 Apr 2010 07:00:00 +0000 Google Annotations Gallery is an exciting new Java open source library that provides a rich set of annotations for developers to express themselves. Do you find the standard Java annotations dry and lackluster? Have you ever resorted to leaving messages to fellow developers with the @Deprecated annotation? Wouldn't you rather leave a @LOL or @Facepalm instead? If so, then this is gallery for you.


Not only can you leave expressive remarks in your code, you can use these annotations to draw attention to your poetic endeavors. How many times have you written a palindromic or synechdochal line of code and wished you could annotate it for future readers to admire? Look no further than @Palindrome and @Synechdoche.

But wait, there's more. The Google Annotations Gallery comes complete with dynamic bytecode instrumentation. By using the gag-agent.jar Java agent, you can have your annotations behavior-enforced at runtime. For example, if you want to ensure that a method parameter is non-zero, try @ThisHadBetterNotBe(Property.ZERO). Want to completely inhibit a method's implementation? Try @Noop.

If we've whet your appetite for truly expressive annotations, then read on and immerse yourself in the Google Annotations Gallery.

By Leo Deegan, Software Engineering Team

Update: Happy April Fool's Day!
]]> https://googledata.org/google-open-source/google-annotations-gallery/feed/ 0 Leuven, Belgium GSoC Infosession https://googledata.org/google-open-source/leuven-belgium-gsoc-infosession/ https://googledata.org/google-open-source/leuven-belgium-gsoc-infosession/#comments Mon, 15 Mar 2010 18:25:00 +0000 KDE and Thousand Parsec), Ruben Vermeersch (K.U. Leuven researcher and GNOME Google Summer of Code admin) and Bram Luyten (@mire co-founder and mentor for DSpace) gave a presentation about the Google Summer of Code 2010 program to an audience of interested students.

The Google Summer of Code schedule is quite challenging for Belgian students because of the large overlaps between the program and their examinations. However, the presenters made it clear that with careful planning in the application, and transparent communication with mentors, successful participation is definitely possible. As an added bonus, if students can find a mentor in a company, participation in Google Summer of Code can be counted as an internship for some of the master's programs at K.U. Leuvens, which adds even more value on top of the stipends.

For many of the attending students, it sounded too good to be true, as we saw true stares of disbelief when the stipend of 5000 USD in exchange for a few months of programming was announced. Vincent's testimonials of his experiences as a student for the KDE and Thousand Parsec projects, and Ruben's recruiting talk for GNOME "Become the Next GNOME Rockstar," convinced them in the end.

The slides, a recording of the event (in Dutch) and additional information is available.

By Bram Luyten, Head of Sales and Marketing, @Mire
]]> https://googledata.org/google-open-source/leuven-belgium-gsoc-infosession/feed/ 0 RE2: a principled approach to regular expression matching https://googledata.org/google-open-source/re2-a-principled-approach-to-regular-expression-matching/ https://googledata.org/google-open-source/re2-a-principled-approach-to-regular-expression-matching/#comments Thu, 11 Mar 2010 20:00:00 +0000 Regular expressions are one of computer science's shining examples of the benefits of good computer science theory. They were originally developed by theorists as a way to describe infinite sets, but Ken Thompson introduced them to programmers as a way to describe text patterns in his implementation of the text editor QED for CTSS. Dennis Ritchie followed suit in his own implementation of QED, for GE-TSS. Thompson and Ritchie would go on to create Unix, and they brought regular expressions with them. By the late 1970s, regular expressions were a key feature of the Unix landscape, in tools such as ed, sed, grep, egrep, awk, and lex. They remain a key feature of the open source landscape today, in those venerable Unix tools and at the core of new languages like Perl, Python, and JavaScript.

The feature-rich regular expression implementations of today are based on a backtracking search with a potential for exponential run time and unbounded stack usage. At Google, we use regular expressions as part of the interface to many external and internal systems, including Code Search, Sawzall, and Bigtable. Those systems process large amounts of data; exponential run time would be a serious problem. On a more practical note, these are multithreaded C++ programs with fixed-size stacks: the unbounded stack usage in typical regular expression implementations leads to stack overflows and server crashes. To solve both problems, we've built a new regular expression engine, called RE2, which is based on automata theory and guarantees that searches complete in linear time with respect to the size of the input and in a fixed amount of stack space.

Today, we released RE2 as an open source project. It's a mostly drop-in replacement for PCRE's C++ bindings
and is available under a BSD-style license. See the RE2 project page for details.

By Russ Cox, Software Engineering Team
]]> https://googledata.org/google-open-source/re2-a-principled-approach-to-regular-expression-matching/feed/ 0 Low-Impact Operating System Tracing https://googledata.org/google-open-source/low-impact-operating-system-tracing/ https://googledata.org/google-open-source/low-impact-operating-system-tracing/#comments Fri, 05 Mar 2010 17:51:00 +0000
The open source projects he created as part of his work were two-fold: Linux Trace Toolkit Next Generation (LTTng), a LGPLv2.1/GPLv2 tracer for the Linux kernel; and Userspace RCU library (liburcu), a highly-scalable user-space synchronization library, distributed under the LGPLv2.1 license.

Mathieu was kind enough to send us this summary of his research:


Computer systems, both at the hardware and software-levels, are becoming increasingly complex. Tracing is the key to solving some or all of this increasing complexity. In the case of Linux, used in a large range of applications, from small embedded devices to high-end servers, the size of the operating system kernels are increasing, libraries are being added, and major redesign of existing software is required to benefit from multi-core architectures. As a result, the software development industry and individual developers are facing problems whose resolution requires an understanding of the interaction between applications and all components of an operating system.

In my thesis, I propose the LTTng (Linux Trace Toolkit next generation) tracer as an answer to the industry and open source community tracing needs. The low-intrusiveness of the tracer is a key aspect of its usefulness because we need to be able to reproduce problems occurring in normal conditions. In some cases, users leave tracers active at all times in production, which makes the tracer overhead definitely critical. Our approach involves the design of synchronization primitives that meet the low-impact requirements. The linearly scalable and wait-free RCU (Read-Copy Update) synchronization mechanism used by the LTTng tracer fulfills these requirements with respect to data read. A custom-made buffer synchronization scheme is proposed to extract tracing data while preserving linear scalability and wait-free characteristics.

By measuring the LTTng impact, I demonstrate that it is possible to create a tracer that satisfy all the following characteristics: low latency, deterministic real-time impact (wait-free), small impact on operating system throughput and linear scalability with the number of cores. Experiments on various architectures show that this tracer is portable.

I propose a general model for superscalar multi-core systems with weakly-ordered memory accesses to perform formal verification of the RCU correctness and wait-free guarantees by model-checking. The LTTng
buffering scheme is also formally verified for safety and progress. Formal verification demonstrates that these algorithms allow reentrancy from multiple execution contexts, ranging from standard thread to non-maskable interrupts handlers, allowing a wide instrumentation coverage of the operating system.


Many thanks to Mathieu for sending us this report. You can download the full dissertation for more details.

By Carol Smith, Open Source Team
]]> https://googledata.org/google-open-source/low-impact-operating-system-tracing/feed/ 0 Life’s a Beach: Google Summer of Code and the Abelian Sandpile Model https://googledata.org/google-open-source/lifes-a-beach-google-summer-of-code-and-the-abelian-sandpile-model/ https://googledata.org/google-open-source/lifes-a-beach-google-summer-of-code-and-the-abelian-sandpile-model/#comments Thu, 04 Feb 2010 22:08:00 +0000 Abelian Sandpile Model (ASM) is a mathematical model of a pile of sand developed by physicists around 1990 to exemplify self-organized criticality, a phenomenon conjecturally ubiquitous in nature. Roughly, self-organized criticality describes a system that naturally evolves into a barely-stable non-equilibrium condition, where the instability is characterized by scale invariance. The Gutenberg-Richter law in geophysics and Zipf's law in linguistics are often cited as real-world examples. More recently, the ASM has been shown to have connections to algebraic geometry, combinatorics, and number theory.

The ASM starts with a graph having a finite number of vertices and edges. One is allowed to place grains of sand on each vertex. If there is enough sand on a vertex, the vertex is allowed to "fire", sending a grain of sand along each out-going edge to its neighboring vertices. These vertices, in turn, may then have enough sand to fire, etc., creating an avalanche. One often designates a "sink" vertex that only absorbs sand. In this case, if each vertex has a edge-path to the sink, any configuration of sand put on the graph will, after a sequence of firings, eventually stabilize. This stable state is independent of the order of the firings.

For his Google Summer of Code™ project, working in consultation with experts in the field, Bryan Head created a state-of-the art Java program to visualize and analyze the ASM. His program includes: a flexible graph editing environment for the creation of sandpiles on arbitrary weighted digraphs; multiple visualization modes in two- and three-dimensions using OpenGL; and an interface to the free open-source mathematical software, Sage, in particular with David Perkinson's Sage Sandpiles software.

This codebase has attracted significant interest from the sandpiles community. Bryan's visualizations are helping researchers develop better mathematical models.



If you are interested in sandpiles, you can download Bryan's program or learn more about ASM.

By: Bart Massey, Google Summer of Code Organization Administrator
]]> https://googledata.org/google-open-source/lifes-a-beach-google-summer-of-code-and-the-abelian-sandpile-model/feed/ 0 Libevent 2.0.x: Like Libevent 1.4.x, Only More So. https://googledata.org/google-open-source/libevent-2-0-x-like-libevent-1-4-x-only-more-so/ https://googledata.org/google-open-source/libevent-2-0-x-like-libevent-1-4-x-only-more-so/#comments Tue, 26 Jan 2010 22:56:00 +0000 Libevent from the Tor Project. Libevent is a software library whose purpose is to provide consistent fast interfaces to various operating systems' mutually incompatible fast networking facilities. gives applications two basic interfaces to these networking layers: a low-level interface where the application is notified when an operation (like a network read or write) is ready to begin, and a higher-level interface where Libevent itself manages network operations and the application is notified when the network operations are completed.

It's amazing how complicated things can become when you set out to do a simple task pretty well.

Back in 2003, I found myself working on a project written in C that needed (among other things) to do fast nonblocking IO on a whole bunch of sockets at once. Our placeholder asynchronous IO implementation, based on the widely portable select() and poll() syscalls, wasn't working well on Windows and didn't scale at all.

What does it do?

I'll add a bit of background here, for readers who haven't tried to do their nonblocking IO from scratch before. When you write a program that needs to work with many active network connections at once, you have two basic choices. One choice is to call IO functions that wait (block) until they are done, and to give each connection its own thread or process so that one connection's blocking doesn't force all the connections to block. For many applications, this approach doesn't scale too well.

A better choice is to use so-called "nonblocking" IO functions that make as much progress as they can, and return control to your program once they can make no more immediate progress. With this approach, your program needs some way to tell which connections are ready to do more IO, so that you don't waste CPU time polling connections that aren't ready yet. There are many kernel-provided APIs to do this, but the portable ones all scale badly (like select() and poll(), which are both O(N) in performance), and the fast ones are all non-portable (like kqueue() and epoll() and /dev/poll). This is where Libevent comes in. Libevent provides a consistent interface to wrap the common features of all these interfaces that tell you when connections are ready for IO, so that you can write clean code using nonblocking IO without having to re-code for every possible backend.

So, we started using Niels Provos's Libevent library around 2005. It was around version 1.0 back then, and it had... a few bugs. To get our application working I started submitting patches, and before too long Niels asked me if I'd like to co-maintain it.

Flash-forward to the present day. Libevent had grown an "extras" library to support commonly needed protocols like HTTP and DNS. Libevent had gotten real users too, including Tor, Memcached, and Chromium. With version 1.4, Libevent's core had grown to maturity, and did a pretty good job on most Unixish platforms and a not-too-awful job on Windows. But there were still some issues that made us decide to try for a major development effort in 2.0. I'll touch on the big ones here, explain what progress we've made, and what we still have to do before we can call Libevent 2.0 finished.

Cleaner interfaces
Some of our older interfaces had grown crufty and difficult to use safely. For example, you sometimes needed to remember to follow up every call to event_set() with a corresponding call to event_base_set(), or you might get strange behavior. Some interfaces were a bad idea to begin with, like the ones that relied on global state.

We could have just changed all our APIs to be less error-prone, but that would have broken all the older programs written to use them. Still, leaving the old APIs exposed by default would make it hard for developers to make sure they were avoiding them. As a compromise, we decided to put the preferred APIs in new headers (such as event2/event.h), and the deprecated APIs in compatibility headers (such as event2/event_compat.h), while retaining the old headers (like the old event.h) as wrappers to include both the new and old APIs.


Nearly every non-Windows OS does nonblocking network IO best with the system I described above, where the program first asks the kernel which sockets are ready to read or write and then asks the kernel to perform as much IO on them as could succeed right now without blocking. On Windows, though, the "tell me which sockets are ready" part is not so efficient or scalable. Windows's select() function (which Libevent 1.4 uses) is O(N), its WSAWaitForMultipleEvents() function doesn't support more than 64 sockets at a time, and so on.

Windows does have a good networking API, however. It just follows a different paradigm. Instead of asking the OS to tell you when an operation would be able to make progress, you tell the OS to begin an operation asynchronously, and ask the OS later (via an , or IOCP) to tell you which operations have finished.

It's quite hard to implement a notify-on-readiness interface using a notify-on-completion system like Windows provides. Fortunately, however, it's pretty easy (and frequently desirable!) to implement a notify-on-completion interface using a Unix-style notify-on-readiness API, so that's what we decided to do.

Supporting IOCP on Windows has been the most far-reaching challenge in Libevent 2.0, and has led to improvements throughout the rest of the (non-Windows-specific) codebase. I'll say more when I talk about individual improvements below.

Though most of the groundwork for an IOCP implementation is laid, the code itself is fairly immature. If you fetch the latest code from subversion, you can enable data transfer through an existing socket with IOCP. IOCP-enhanced connect and accept operations are not yet supported, nor is UDP. We hope to get these done pretty soon.


Buffered-IO improvements

Earlier Libevent versions had a "bufferevent" abstraction to handle the common case where you want to queue data received and data to be sent over a TCP connection, and receive notification as more data arrives or is sent. The interface here was a good match for IOCP, but the existing implementation was too inflexible and inefficient for serious use. Among other problems, its storage format relied on large circular buffers, it allowed only a single implementation for the interface, it behaved poorly on Windows, and it was just generally slow.


In Libevent 2.0, we've rewritten bufferevent from the ground up. The data storage format has changed from circular buffers to a linked list of buffers, reminiscent of the classic mbuf interface. We've added support for sending files with sendfile() and friends, and fixed up Windows support. The interface now supports multiple backends, OO-style, to allow multiple implementations of the "buffered IO" API. So far, we have five such backends: the basic socket implementation, a filtering wrapper implementation, an SSL-based implementation (see below), a socketpair-style implementation, and an IOCP-based implementation.

Preliminary results around April 2009 indicated that the rewritten codebase had improved our large-payload HTTP benchmark performance by between 35 and 174 percent. We hope that as we profile and benchmark this code more aggressively, we can find more opportunities for improvement here.


Improved multithreading

Older versions of Libevent allowed one event loop structure per thread; adding or deleting events from another thread wasn't supported. This was not only an annoyance for writers of multithreaded applications; it made IOCP support nigh-impossible, since we needed to have the IOCP loop run in separate threads from the main event thread.

The basic work here is finally done; as of last week, there are no major non-locked parts of Libevent remaining. We may still need to do performance tuning here: different operating systems provide mutex implementations with wildly differing characteristics.


Our next big multithreading initiative will be better support for parallelizing event callbacks across multiple CPUs on demand. This will be trivial for Win32 IOCP callbacks, but will require actual coding for older Unix backends. To ease migration, we'll need to allow programs to enable this functionality one callback at a time, so that we don't force them to upgrade all of their code at once.


As noted above, there's now a bufferevent backend that uses the OpenSSL library to encrypt traffic using the SSL/TLS family of protocols. This wasn't as simple as writing a regular data filter, since the protocol's support for session renegotiation allows a single virtual read/write operation to be implemented as multiple reads and writes to the network.


The OpenSSL cryptography library has support for doing TLS encryption directly to the network or over a user-provided IO object. We've taken both approaches, so that an OpenSSL bufferevent can either deliver data to a network socket or to another bufferevent. This way, Unix programs can have a bufferevent that speaks SSL directly to the network, and Windows programs can wrap an SSL layer over an IOCP-based backend.

We'd eventually like to add support for other free SSL libraries, such as GnuTLS and NSS, though we're not planning to work on this for Libevent 2.0.x unless somebody wants to contribute the code.

As you might imagine, we've added and rewritten a lot of code between Libevent 1.4 and Libevent 2.0. Without good unit tests, we'd probably have introduced a fair number of bugs.


Unfortunately, our old unit tests weren't so good. Though the overall coverage was decent (about 72%), some parts of the code were hardly tested at all (the DNS code and the IO buffer code were both under 60% coverage). The test suite was fairly fragile, too: many tests relied on global state, which made it hard to write new ones, especially if they required inducing strange error conditions.


For Libevent 2.0, we put a major effort into our testing code. Tests now run in separate processes, so they can't affect one another no matter how badly they trash global state. This not only stopped us from adding new bugs, but also revealed a few longstanding ones. Thanks to this change, it's easy to test far more error conditions than we could test before. This has gotten us up to around 80% test coverage overall, with the least-tested module at around 72%.

Before we release Libevent 2.0, I want our test coverage to be at least 80% for every module. I'd also like to do a manual coverage audit to look for any untested sections that seem particularly error-prone.



Next steps
In the coming months, we'll want to get the Libevent 2.0.x series finished. This will mainly involve fixing known bugs in the Sourceforge trackers, finishing the IOCP code, and testing the heck out of everything.

In parallel, I've been working on a short book to serve as an introduction to nonblocking networking in general, and to Libevent in particular. The latest draft is at http://www.wangafu.net/~nickm/libevent-book/.

After Libevent 2.0.x is stable and released, we'll be able to start looking at missing features for Libevent 2.1.x. These aren't at all final yet, but I'm hoping to get improved thread-pool support for IOCP and non-IOCP users alike, and better support for UDP-based protocols. We'll probably have many other changes to make based on user feedback from Libevent 2.0.

Thanks to Google for their support of the project.


By Nick Mathewson, Tor Project
]]> https://googledata.org/google-open-source/libevent-2-0-x-like-libevent-1-4-x-only-more-so/feed/ 0