Running Karma tests on BrowserStack

Browserstack is a great setup and Karma is a great test runner for javascript, so the two should be a pretty good fit right? In practice it took me quite a lot of time to get the settings right, including discussions on the karma users group and some help from the good folks at BrowserStack. What follows are some brief notes on how I set this up to run with Grunt and on the Jenkins build server.

Karma

The karma-browserstack-launcher plugin covers pretty well how to configure your browserstack account and add custom launchers. What it does not cover is

1. How to get correct values to use for setting up the custom launchers.
   Turns out this is quite easy, and is covered by the REST API pages on Browserstack. You get a full list of all possible values by issuing

    curl -u "my_user:0x_my_hash_xyz" https://www.browserstack.com/automate/browsers.json

2. Dealing with the flakey connection to BrowserStack and its slow virtual machines.
   You can find a lot of Stack Overflow on the issue of warnings on DISCONNECTED browsers, and this very much applies to BrowserStack. Running a test on browserstack can mean that you first fire up virtual machine running Windows, then fire up a simulator running Android, and finally run the javascript tests. This of course takes time – and a lot of the time a lot more time than the default settings of Karma permit. Sometimes we are talking minutes. Recent versions of Karma has gained a few new options and I have the following options in my karma.config.js:

   // to avoid DISCONNECTED messages when connecting to BrowserStack
   browserDisconnectTimeout : 10000, // default 2000
   browserDisconnectTolerance : 1, // default 0
   browserNoActivityTimeout : 4*60*1000 //default 10000
   captureTimeout : 4*60*1000 //default 60000

3. what to do when hitting limits on the number of concurrent runs.
   An account has various limits on how many concurrent test runs you can run, so sometimes the browsers are disconnected by Karma before its their turn. The easy fix is simply to split the test runs into smaller batches. Each group of browser gets one config file each. My solutions was to simply have one main minimal configuration file where I define the custom launchers, but no browser section. I then <code>require</code> this into the various browser configurations. This also allows me to have separate output files for each browser, which is useful in combination with Jenkins to show a graph of the number of successful and failed tests per browser.

   // test/karma/browserstack/chrome.karma.config.js
   var commonBrowserStackSetup = require('./_common.conf.js');
   module.exports = function (config) {
     commonBrowserStackSetup(config);
     config.set({
       port : 9894, // different port for each browser group
       browsers : ['bs_chrome_34', 'bs_chrome_31'],
       junitReporter: { outputFile: 'test/test-results/test-results-chrome.xml'}
     });
   };

4. Dealing with tests timing out after 2000 milliseconds
   Even after removing the test specific timeouts, some tests might still run too slow on the Android emulators, and might actually time out after the default setting of two seconds. When running mocha on the command line, one can override the default timeout setting, and it turns out we can do it in the configuration as well. Simply add the following to your config to increase the default timeout.

   client: {
     mocha: {
       timeout : 20000 // 20 seconds
     }
   }

   I only add this to my BrowserStack config, as I usually want the tests to time out long before 2 seconds have passed to alert me that I am doing something wrong.

My final karma config directory layout looks like this, and I can combine configurations like shown above.

test/karma/          # all karma config
|-- _coverage.js     # coverage reporting
|-- browserstack     # browserstack specific
| |-- _common.conf.js# common setup for BS
| |-- android.js     # android browser tests
| |-- chrome.js      # ...
| |-- firefox.js
| |-- ie.js
| |-- ios.js
| |-- safari.js
`-- karma.minimal.conf.js # minimal config

Grunt

While Grunt is not needed for running Karma tests, I need it to build the project, and then I might as well make it run the tests as well 🙂 The relevant part of my Grunt karma config are as follows:

karma : {
  // due to our subscription only allowing
  // two parallel test runs, we need to batch the tests
  browserStack_ie : { configFile : 'test/karma/browserstack/ie.js' },
  browserStack_ff : { configFile : 'test/karma/browserstack/ff.js' },
  browserStack_safari : { configFile : 'test/karma/browserstack/safari.js' },
  browserStack_ios : { configFile : 'test/karma/browserstack/ios.js' },
  browserStack_android : { configFile : 'test/karma/browserstack/android.js' },
  browserStack_chrome: { configFile : 'test/karma/browserstack/chrome.js' }
}

And the task definition

// browserstack-tests - due to concurrency limits, run sequentially
grunt.registerTask('browserstack', [
 'karma:browserStack_ie',
 'karma:browserStack_ios',
 'karma:browserStack_android',
 'karma:browserStack_ff',
 'karma:browserStack_safari',
 'karma:browserStack_chrome'
]);

All the tests can now be run like this

grunt browserstack

Jenkins

Setting up the Jenkins build is rather easy. The only complicating factor is that due to the slowness of BrowserStack, all test specific timeouts are out. I still want them when I develop on my local machine, though, and so I opted to just remove them on the Jenkins CI Server. To run the tests simply add a build step «Execute shell» and input something like the following:

# remove test specific timeouts overrides - problematic due to slow vms on browserstack
find test/ -name '*.js' -exec sed -i 's/.*this\.timeout.*//' {} \;
npm install
# build and test project. --force makes all tests run regardless whether they fail
grunt --no-color build && grunt --no-color --force browserstack

Additional resources

• Article from the Karma devs on setting up Jenkins
• Debugging DISCONNECTED messages