Yet Another JavaScript Testing Framework
May 22nd, 2009 | Published in Google Testing
by Miško Hevery & Jeremie Lenfant-engelmann
Did you notice that there are a lot of JavaScript testing frameworks out there? Why has the JavaScript community not consolidated on a single JavaScript framework the way Java has on JUnit. My feeling is that all of these frameworks are good at something but none solve the complete package. Here is what I want out of JavaScript unit-test framework:
I don't know of any JavaScript framework out there which will let me do what I want. In order to achieve my goal I need a JavaScript framework with these three features:
Command Line Control
Most JavaScript test runners consist of JavaScript application which runs completely in the browser. What this means in practice is that I have to go to the browser and refresh the page to get my results. But browsers need an HTML file to display, which means that I have to write HTML file which loads all of my production code and my tests before I can run my tests. Now since browsers are sandboxes, the JavaScript tests runner can only report the result of the test run inside the browser window for human consumption only. This implies that 1) I cannot trigger running of the tests by hitting Ctrl-S in my IDE, I have to Alt-tab to Browser, hit refresh and Alt-tab back to the IDE and 2) I cannot display my test result in my IDE, the results are in the browser in human readable form only.
On my continuous build machine I need to be able to run the same tests and somehow get the failures out of the browser and on to the status page. Most JavaScript test runners have a very poor story here, which makes integrating them into a continuous build very difficult.
What we need, is the ability to control test execution from command line so that I can trigger it from my IDE, or my continuous build machine. And I need test failures to be reported on the command line (not inside the browser where they are unreachable) so that I can display them in IDE or in continuous build status page.
Parallel Execution
Since most JavaScript test runners run fully in the browser I can only run my test on one browser at a time during my development process. In practice this means that you don't find out about failures in other browser until you have checked in the code to your continuous build machine (if you were able to set it up) and your code executes on all browsers. By that point you have completely forgotten about what you have written and debugging becomes a pain. When I run my test I want to run them on all browser platforms in parallel.
Instant Feedback in IDE
After I hit Ctrl-S on my IDE, my patience for test results is about two seconds before I start to get annoyed. What this means in practice is that you can not wait until the browser launches and runs the tests. The browser needs to be already running. Hitting refresh on your browser manually is very expensive since the browser needs to reload all of the JavaScript code an re-parse it. If you have one HTML file for each TestCase and you have hundred of these TestCases, The browser may be busy for several minutes until it reloads and re-parses the same exact production code once for each TestCase. There is no way you can fit that into the patience of average developer after hitting Ctrl-S.
Introducing JsTestDriver
Jeremie Lenfant-engelmann and I have set out to build a JavaScript test runner which solves exactly these issues so that Ctrl-S causes all of my JavaScript tests to execute in under a second on all browsers. Here is how Jeremie has made this seemingly impossible dream a reality. On startup JsTestDriver captures any number of browsers from any number of platforms and turns them into slaves. As slave the browser has your production code loaded along with all of your test code. As you edit your code and hit Ctrl-S the JsTestDriver reloads only the files which you have modified into the captured browsers slaves, this greatly reduces the amount of network traffic and amount of JavaScript re-parsing which the browser has to do and therefore greatly improves the test execution time. The JsTestDriver than runs all of your test in parallel on all captured browsers. Because JavaScript APIs are non-blocking it is almost impossible for your tests to run slow, since there is nothing to block on, no network traffic and no re-parsing of the JavaScript code. As a result JsTestDriver can easily run hundreds of TestCases per second. Once the tests execute the results are sent over the network to the command which executed the tests either on the command line ready to be show in you IDE or in your continuous build.
Demo
Did you notice that there are a lot of JavaScript testing frameworks out there? Why has the JavaScript community not consolidated on a single JavaScript framework the way Java has on JUnit. My feeling is that all of these frameworks are good at something but none solve the complete package. Here is what I want out of JavaScript unit-test framework:
I want to edit my JavaScript code in my favorite IDE, and when I hit Ctrl-S, I want all of the tests to execute across all browsers and return the results immediately.
I don't know of any JavaScript framework out there which will let me do what I want. In order to achieve my goal I need a JavaScript framework with these three features:
Command Line Control
Most JavaScript test runners consist of JavaScript application which runs completely in the browser. What this means in practice is that I have to go to the browser and refresh the page to get my results. But browsers need an HTML file to display, which means that I have to write HTML file which loads all of my production code and my tests before I can run my tests. Now since browsers are sandboxes, the JavaScript tests runner can only report the result of the test run inside the browser window for human consumption only. This implies that 1) I cannot trigger running of the tests by hitting Ctrl-S in my IDE, I have to Alt-tab to Browser, hit refresh and Alt-tab back to the IDE and 2) I cannot display my test result in my IDE, the results are in the browser in human readable form only.
On my continuous build machine I need to be able to run the same tests and somehow get the failures out of the browser and on to the status page. Most JavaScript test runners have a very poor story here, which makes integrating them into a continuous build very difficult.
What we need, is the ability to control test execution from command line so that I can trigger it from my IDE, or my continuous build machine. And I need test failures to be reported on the command line (not inside the browser where they are unreachable) so that I can display them in IDE or in continuous build status page.
Parallel Execution
Since most JavaScript test runners run fully in the browser I can only run my test on one browser at a time during my development process. In practice this means that you don't find out about failures in other browser until you have checked in the code to your continuous build machine (if you were able to set it up) and your code executes on all browsers. By that point you have completely forgotten about what you have written and debugging becomes a pain. When I run my test I want to run them on all browser platforms in parallel.
Instant Feedback in IDE
After I hit Ctrl-S on my IDE, my patience for test results is about two seconds before I start to get annoyed. What this means in practice is that you can not wait until the browser launches and runs the tests. The browser needs to be already running. Hitting refresh on your browser manually is very expensive since the browser needs to reload all of the JavaScript code an re-parse it. If you have one HTML file for each TestCase and you have hundred of these TestCases, The browser may be busy for several minutes until it reloads and re-parses the same exact production code once for each TestCase. There is no way you can fit that into the patience of average developer after hitting Ctrl-S.
Introducing JsTestDriver
Jeremie Lenfant-engelmann and I have set out to build a JavaScript test runner which solves exactly these issues so that Ctrl-S causes all of my JavaScript tests to execute in under a second on all browsers. Here is how Jeremie has made this seemingly impossible dream a reality. On startup JsTestDriver captures any number of browsers from any number of platforms and turns them into slaves. As slave the browser has your production code loaded along with all of your test code. As you edit your code and hit Ctrl-S the JsTestDriver reloads only the files which you have modified into the captured browsers slaves, this greatly reduces the amount of network traffic and amount of JavaScript re-parsing which the browser has to do and therefore greatly improves the test execution time. The JsTestDriver than runs all of your test in parallel on all captured browsers. Because JavaScript APIs are non-blocking it is almost impossible for your tests to run slow, since there is nothing to block on, no network traffic and no re-parsing of the JavaScript code. As a result JsTestDriver can easily run hundreds of TestCases per second. Once the tests execute the results are sent over the network to the command which executed the tests either on the command line ready to be show in you IDE or in your continuous build.
Demo