Google Data » Alberto Savoia

This Code is CRAP

Alberto Savoia — Tue, 22 Feb 2011 21:45:00 +0000

Note: This post is rated PG-13 for use of a mild expletive. If you are likely to be offended by the repeated use a word commonly heard in elementary school playgrounds, please don’t read any further.

CRAP is short for Change Risk Anti-Patterns – a mildly offensive acronym to protect you from deeply offensive code. CRAP was originally developed and launched in 2007 by yours truly (Alberto Savoia) and my colleague and partner in crime Bob Evans.

Why call it CRAP? When a developer or tester has to work with someone else’s (bad) code, they rarely comment on it by saying things like: “The median cyclomatic complexity is unacceptable,” or “The efferent coupling values are too high.” Instead of citing a particular objective metric, they summarize their subjective evaluation and say things like: “This code is crap!” At least those are the words the more polite developers use; I’ve heard and read far more colorful adjectives and descriptions over the years. So Bob and I decided to coin an acronym that, in addition to being memorable – even if it’s for the wrong reasons, is a good match with the language that its intended users use and it’s guaranteed to grab a developer’s attention: “Hey, your code is CRAP!”

But what makes a particular piece of code CRAP? There is, of course, no fool-proof, 100% objective, and accurate way to determine CRAPpiness. However, our experience and intuition – backed by a bit of research and a lot of empirical evidence – suggested the possibility that there are detectable and measurable patterns that indicate the possible presence of CRAPpy code. That was enough to get us going with the first anti-pattern (which I’ll describe shortly.)

Since its inception, the original version of CRAP has gained quite a following; it has been ported to various languages and platforms (e.g. Java, .NET, Ruby, PHP, Maven, Ant) and it’s showing up both in free and commercial code analysis tools such as Hudson’s Cobertura and Atlassian’s Clover. Do a Google search for “CRAP code metric” and you’ll see quite a bit of activity. All of which is making Bob and I feel mighty proud, but we haven’t been resting on our laurels. Well, actually we have done precisely that. After our initial work (which included the Crap4J Eclipse plug-in and the, now mostly abandoned, crap4j.org website) we both went to work for Google and got busy with other projects. However, the success and adoption of CRAP is a good indication that we were on to something and I believe it’s time to invest a bit more in it and move it forward.

Over the next few weeks I will post about the past, present and future of CRAP. By the time I’m done, you will have the tools to:

- Know you CRAP
- Cut the CRAP, and
- Don’t take CRAP from nobody!

I’ll finish today’s entry with a bit of background on the original CRAP metric.

A Brief History of CRAP

As the CRAP acronym suggests, there are several possible patterns that make a piece of code CRAPpy, but we had to start somewhere. Here is the first version of the (in)famous formula to help detect CRAPpy Java methods. Let’s call it CRAP1, to make clear that this covers just one of the many interesting anti-patterns and that there are more to come.

CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)

Where CRAP1(m) is the CRAP1 score for a method m, comp(m) is the cyclomatic complexity of m, and cov(m) is the basis path code coverage from automated tests for m.

If CRAP1(m) > 30, we consider the method to be CRAPpy.

This CRAP1 formula did not materialize out of thin air. We arrived at this particular function empirically; it’s the result of a best fit curve achieved through a lot of trial-and-error. At the time we had access to the source code for a large number of open source and commercial Java projects, along with their associated JUnit tests. This allowed us to rank code for CRAPpiness using one formula, ask our colleagues if they agreed and kept iterating until we reached diminishing returns. This way we were able to come up with a curve that was a pretty good fit for the more subjective data we got from our colleagues.

Here’s why we think that CRAP1 is a good anti-pattern to detect. Writing automated tests (e.g., using JUnit) for complex and convoluted code is particularly challenging, so crappy code usually comes with few, if any, automated tests. This means that the presence of automated tests implies not only some degree of testability (which in turn seems to be associated with better, or more thoughtful, design), but it also means that the developers cared enough, knew enough and had enough time to write tests – which is another good sign for the people inheriting the code. These sounded like reasonable assumptions at the time, and the adoption of CRAP1 – especially by the Agile community – reflects that.

Like all software metrics, CRAP1 is neither perfect nor complete. We know very well, for example, that you can have great code coverage and lousy tests. In addition, sometimes complex code is either unavoidable or preferable; there might be instances where a single higher complexity method might be easier to understand than three simpler ones. We are also aware that the CRAP1 formula doesn’t currently take into account higher-order, more design-oriented metrics that are relevant to maintainability (such as cohesion and coupling) – but it’s a start, the plan is to add more anti-patterns.

Use CRAP On Your Project

Even though Bob and I haven't actively developed or maintained Crap4J in the past few years (shame on us!), other brave developers have been busy porting CRAP to all sorts of languages and environments. As a result, there are many versions of the CRAP metric in open source and commercial tools. If you want to try CRAP on your project, the best thing to do is to run a Google search for the language and tools you are currently using.

For example, a search for "crap metric .net" returned several projects, including crap4n and one called crap4net. If you use Clover, here's how you can use it to implement CRAP. PHP? No problem, someone implemented CRAP for PHPUnit. However, apparently nobody has implemented CRAP for COBOL yet ... here's your big chance!

Until the next blog on CRAP, you might enjoy this vintage video on Crap4J. Please note, however, that the Eclipse plug-in shown in the demo does not work with versions of Eclipse newer than 3.3 - we did say it was a vintage video and that Bob and I have been resting on our laurels!

Posted by Alberto Savoia

Google Testing Blog 2010-09-27 11:12:00

Alberto Savoia — Mon, 27 Sep 2010 16:12:00 +0000

Update! Speakers & Talks for GTAC 2010

We are thrilled to announce the speakers and talks for the 5th Google Test Automation Conference (GTAC). This year’s event will have a total of 11 talks. This includes the three keynotes that we announces earlier and eight other talks. These talks span the three sub-categories of Testing, Testability and Test Automation which are an integral part of this year’s theme “Test to Testability”.

As we had shared earlier, for this year’s GTAC we used a new process of letting the selected attendees vote on the talks they wanted be a part of GTAC. The committee tallied the votes and ensures a healthy distribution between topics and participants from across the globe and relevance to our theme. We received over 80 submissions and have an acceptance rate of about 10%. Our thanks to everyone who submitted a proposal and all the attendees who voted to make this a successful process.

Here is the list of talks. More details can be found at the select talks page on the GTAC site.

Category: Testing

Early Test Feedback by Test Prioritisation (Shin Yoo, University College London & Robert Nilsson, Google Inc.)
Crowd-source testing, Mozilla community style (Matt Evans, Mozilla)
Measuring and Monitoring Experience in Interactive Streaming Multimedia Web Applications (Shreeshankar Chatterjee, Adobe Systems India)

Category: Testability

Flexible Design? Testable Design? You Don’t Have To Choose! (Russ Rufer and Tracy Bialik, Google Inc.)
Git Bisect and Testing (Christian Couder)
Lessons Learned from Testability Failures (Esteban Manchado Velazquez, Opera Software ASA)

Category: Test Automation

The Future of Front-End Testing (Greg Dennis and Simon Stewart, Google Inc.)
Twist, a next generation functional testing tool for building and evolving test suites (Vivek Prahlad, ThoughtWorks)

For further information on the conference please visit its webpage at http://www.gtac.biz.

Sujay Sahni for the GTAC 2010 Committee

Code coverage goal: 80% and no less!

Alberto Savoia — Tue, 20 Jul 2010 16:13:00 +0000

by Alberto Savoia

I first posted this article a few years ago on the Artima Developer website; but the question of what's adequate code coverage keeps coming up, so I thought it was time for a repost of Testivus wisdom on the subject.

Testivus on Test Coverage

Early one morning, a young programmer asked the great master:

“I am ready to write some unit tests. What code coverage should I aim for?”

The great master replied:

“Don’t worry about coverage, just write some good tests.”

The young programmer smiled, bowed, and left.

Later that day, a second programmer asked the same question.

The great master pointed at a pot of boiling water and said:

“How many grains of rice should I put in that pot?”

The programmer, looking puzzled, replied:

“How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.”

“Exactly,” said the great master.

The second programmer smiled, bowed, and left.

Toward the end of the day, a third programmer came and asked the same question about code coverage.

“Eighty percent and no less!” Replied the master in a stern voice, pounding his fist on the table.

The third programmer smiled, bowed, and left.

After this last reply, a young apprentice approached the great master:

“Great master, today I overheard you answer the same question about code coverage with three different answers. Why?”

The great master stood up from his chair:

“Come get some fresh tea with me and let’s talk about it.”

After they filled their cups with smoking hot green tea, the great master began:

“The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.
The second programmer, on the other hand, is quite experienced both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.”

“I see,” said the young apprentice, “but if there is no single simple answer, then why did you tell the third programmer ‘Eighty percent and no less’?”

The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down.

“The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.”

The young apprentice and the grizzled great master finished drinking their tea in contemplative silence.

Testivus, Testability and Dr. Jekyll and Mr. Hyde

Alberto Savoia — Thu, 01 Jul 2010 17:45:00 +0000

by Alberto Savoia

Note: Apparently, there were lots of downloads of the Testivus booklet and I hit some kind of quota on my personal account. If you have problems with reaching the original link below, please try this new download link or this one.

A major topic at this year's GTAC conference is going to be testability: "We also want to highlight methodologies and tools that can be used to build testability into our products." That's great!

Testability is one of the most important, yet overlooked, attributes of code – and one that is not discussed enough. That's unfortunate, because by the time the issue of testability comes up in a project it's usually too late. As preparation and seeding for GTAC, I though it would be fun and useful to get some discussions on testability going. So here we go, feel free to chime in with your thoughts.

A few years ago, after watching one too many episodes of Kung Fu, I was inspired to write a pretentious and cryptic little booklet about testing called "The Way of Testivus" (PDF).

Testivus addresses the issue of testability in a few places, but I would like to start the discussion with this maxim:

To me, "Think of code and tests as one" is the very foundation of testability. If you don't think about testing as you design and implement your code, you are very likely to make choices that will impair testability when the time comes. This position seemed obvious and non-controversial to me at the time I wrote it, and I still stand by it. Most people seem to agree with it as well, and more than one person told me that it's their favorite and most applicable maxim from all of Testivus. There are however three groups of people who found issue with it.

Some of the people, mostly from the TDD camp, think that my choice of words leaves too much wiggle room: "Thinking about the tests is not enough, they should be writing and running those tests at the same time."

Others think that code and tests should not be thought of as one at all, but they should be treated independently – ideally as adversaries: "I don't want code and tests to be too "friendly". Production code should not be changed or compromised to make the testing easier, and tests should not trust the hooks put in the code to make it more testable." Most of the people in this camp are not big fans of unit/developer testing in the first place, but not all. One person, a believer in developer testing, told me that he gets the best results with a Dr. Jekyll and Mr. Hyde approach. He assumes two different roles and personalities based on whether he's coding or testing his own code. When coding, he's the constructive Dr. Jekyll who focuses on elegant and efficient design and algorithms – and does not worry about testability. When testing, he turns into the destructive Mr. Hyde; he tries to forget that it's his code or how he implemented it, and puts all his energy and anger into trying to break it. Sounds like it could work quite well – though I don't think I'd want this person as an office mate during the Mr. Hyde phase.

A third group, thought that the maxim was fine for unit tests, but not applicable to other types of tests that were best served by an adversarial black-box approach.

What are your thoughts? Is it enough to think about testability when designing or writing the code, or must you actually write and run some tests in parallel with the code? Does anyone agree with the position that code and tests should be designed and developed in isolation? Are there other Dr. Jekylls and Mr. Hydes out there?

Alberto