E2E testing in a Vue.js application

I’ve been working on a Vue.js application, and I was thinking how I can test it easily. The app have a couple of interesting libraries with custom directives and widgets, some of them with scoped-slot. Additionally I have 10-12 Vue components with hierarchical structure in differents .vue files, wich means I need to test features that (as usual) implies > 1 component. And in this case, I have 4 stores.

My first approach was using Jest, it is pretty easy to install and configure. I tried with 2 small and simpliest tests in the beginning and the result was good. But when I started to think to test larger units with 2 or more components, adding scoped-slots + store + custom libraries with custom components +… each tests was incredibly big and unmantainable. Everything for a really small application with in 10-12 components. It doesn’t makes sense to me. And an important issue: scoped-slot only can be emulated, but you can’t add those to the tests.

Second and great option was create E2E tests with Nightwatch.js. I installed it and configure locally and with BrowserStack pretty fast which allow me test on my Firefox and Chrome local installation but most important run my tests on different OS (like Windows and MacOS) and Browsers (like Chrome, Firefox, Safari), but most important I could create tests really easy. Let me show quickly how you can configure it locally on my laptop (Ubuntu 19.04) (installation process is available in docs):

In my root path I have this:

In test folder I have my tests, in conf my nightwatch config files (for BrowserStack and local), in output I have the JUnit XML files (those are results of Selenium tests). If some of the tests doesn’t result as expected, an screenshot is saved in screenshots folder. And finally in log folder there are log files with issues about tests and selenium server results.

Let’s see how looks my local nightwatch settings:

module.exports = {
    src_folders: ["e2e_tests/tests"],
    selenium: {
        launch_url: process.env.HOST,
        start_process: true,
        server_path: require('selenium-server').path,
        cli_args: {
            'webdriver.gecko.driver': require('geckodriver').path,
            'webdriver.chrome.driver': require('chromedriver').path
        },
        log_path: './e2e_tests/log'
    },
    test_settings: {
        default: {
            launch_url: process.env.HOST_TEST,
            screenshots : {
                enabled : true,
                on_failure : true,
                on_error : false,
                path : "./e2e_tests/screenshots"
            },
            output_folder: "./e2e_tests/output"
        },
        chrome: {
            desiredCapabilities : {
                browserName : 'chrome',
                alwaysMatch: {
                    acceptInsecureCerts: true
                },
                chromeOptions: {
                    w3c: false
                }
            },
            webdriver: {
                port: 4444,
                start_process: true,
                server_path: require('chromedriver').path,
                log_path: './e2e_tests/log'
            }
        },
        firefox: {
            desiredCapabilities : {
                browserName : 'firefox',
                alwaysMatch: {
                    acceptInsecureCerts: true
                }
            },
            webdriver: {
                start_process: true,
                port: 4444,
                server_path: require('geckodriver').path,
                log_path: './e2e_tests/log'
            }
        }
    }
}

This settings will open my local Chrome and Firefox browsers and run tests in port 4444.
Just to test the app quickly I can create a simple test in e2e_tests/test folder like:

// e2e_tests/tests/FirstTest.js
module.exports = {
    'Checking title must contains Hello World! string in HTML title' : function (browser) {
        browser
            .url(process.env.HOST)
            .waitForElementVisible('body')
            .assert.titleContains('Hello World!')
            .end();
    }
};

If I add this script in my package.json
"e2e": "./node_modules/.bin/nightwatch e2e_tests/tests/ -c ./e2e_tests/conf/nightwatch.conf.js"
I can run my first test on Chrome and Firefox running: $ npm run e2e -- --env firefox,chrome .
And tatan!! after open both browsers simultaneously and then close automatically, I got something like:

firefox [App] Test Suite
firefox ================
firefox - Connecting to localhost on port 4444…
firefox
firefox ℹ Connected to localhost on port 4444 (7131ms).
firefox Using: firefox (77.0.1) on linux 5.3.0-59-generic platform.
firefox Results for: Checking title must contains Hello World! string
firefox ✔ Element was visible after 142 milliseconds.
firefox ✔ Testing if the page title contains 'Hello World!' (10ms)
firefox ✔ firefox [App] Checking title must contains Hello World! string (5.429s)
firefox [Filter] Test Suite
firefox ===================

chrome [App] Test Suite
chrome ================
chrome - Connecting to localhost on port 4444…
chrome
chrome ℹ Connected to localhost on port 4444 (2494ms).
chrome Using: chrome (83.0.4103.116) on Linux platform.
chrome Results for: Checking title must contains Hello World! string
chrome ✔ Element was visible after 58 milliseconds.
chrome ✔ Testing if the page title contains 'Hello World!' (12ms)
chrome ✔ chrome [App] Checking title must contains Hello World! string (1.944s)
chrome [Filter] Test Suite
chrome ===================

OK. 2 total assertions passed (6.023s)

Now let’s see how looks my nightwatch config file for BrowserStack, so we can run our tests in multiple browsers and OS remotely:

// e2e_tests/conf/browserstack.nightwatch.conf.js

nightwatch_config = {
    src_folders: ["e2e_tests/tests"],
    selenium: {
        start_process: false,
        host: "hub-cloud.browserstack.com",
        port: 80,
        log_path: './e2e_tests/log',
        output_folder: "./e2e_tests/output"
    },
    test_settings: {
        default: {
            launch_url: process.env.HOST,
            screenshots : {
                enabled : true,
                on_failure : true,
                on_error : false,
                path : "./e2e_tests/screenshots"
            },
            log_path: './e2e_tests/log',
            output_folder: "./e2e_tests/output"
        },
        chrome_win10: {
            desiredCapabilities : {
                'browserstack.user': process.env.BROWSERSTACK_USER,
                'browserstack.key': process.env.BROWSERSTACK_ACCESS_KEY,
                'os': 'Windows',
                'os_version': '10',
                'browser': 'Chrome',
                'browser_version': (process.env.CHROME_W10_VERSION) ? process.env.CHROME_W10_VERSION : '83.0',
                'resolution': '1280x800'
            },
            webdriver: {
                log_path: './e2e_tests/log'
            },
            log_path: './e2e_tests/log',
            output_folder: "./e2e_tests/output"
        },
        chrome_catalina: {
            desiredCapabilities: {
                'browserstack.user': process.env.BROWSERSTACK_USER,
                'browserstack.key': process.env.BROWSERSTACK_ACCESS_KEY,
                'os': 'OS X',
                'os_version': 'Catalina',
                'browser': 'Chrome',
                'browser_version': (process.env.CHROME_CATALINA_VERSION) ? process.env.CHROME_CATALINA_VERSION :  '83.0',
                'resolution': '1280x800'
            },
            webdriver: {
                log_path: './e2e_tests/log'
            },
            log_path: './e2e_tests/log',
            output_folder: "./e2e_tests/output"
        },
        firefox_win10: {
            desiredCapabilities : {
                'browserstack.user': process.env.BROWSERSTACK_USER,
                'browserstack.key': process.env.BROWSERSTACK_ACCESS_KEY,
                'os': 'Windows',
                'os_version': '10',
                'browser': 'Firefox',
                'browser_version': (process.env.FIREFOX_W10_VERSION) ? process.env.FIREFOX_W10_VERSION : '77.0',
                'resolution': '1280x800'
            },
            webdriver: {
                log_path: './e2e_tests/log'
            },
            log_path: './e2e_tests/log',
            output_folder: "./e2e_tests/output"
        },
        firefox_catalina: {
            desiredCapabilities : {
                'browserstack.user': process.env.BROWSERSTACK_USER,
                'browserstack.key': process.env.BROWSERSTACK_ACCESS_KEY,
                'os': 'OS X',
                'os_version': 'Catalina',
                'browser': 'Firefox',
                'browser_version': (process.env.FIREFOX_CATALINA_VERSION) ? process.env.FIREFOX_CATALINA_VERSION : '77.0',
                'resolution': '1280x800'
            },
            webdriver: {
                log_path: './e2e_tests/log'
            },
            log_path: './e2e_tests/log',
            output_folder: "./e2e_tests/output"
        },
        safari_catalina: {
            desiredCapabilities: {
                'browserstack.user': process.env.BROWSERSTACK_USER,
                'browserstack.key': process.env.BROWSERSTACK_ACCESS_KEY,
                'os': 'OS X',
                'os_version': 'Catalina',
                'browser': 'Safari',
                'browser_version': (process.env.SAFARI_CATALINA_VERSION) ? process.env.SAFARI_CATALINA_VERSION : '13.0',
                'resolution': '1280x800'
            },
            webdriver: {
                log_path: './e2e_tests/log'
            },
            log_path: './e2e_tests/log',
            output_folder: "./e2e_tests/output"
        }
    }
};

// Code to copy seleniumhost/port into test settings
for(var i in nightwatch_config.test_settings){
    var config = nightwatch_config.test_settings[i];
    config['selenium_host'] = nightwatch_config.selenium.host;
    config['selenium_port'] = nightwatch_config.selenium.port;
}

module.exports = nightwatch_config;

In this file config will run my tests in BrowserStack on Chrome and Firefox in Windows10 and Catalina OS (Mac) and Safari on Catalina OS. Results of those tests will be in e2e_tests/output, but in BrowserStack App Live Dashboard as well with a nice Video and Screenshots. Awesome, eh?

In my package.json I added other script for BrowserStack tests: "e2e:browserstack": "node ./e2e_tests/conf/local.runner.js -c ./e2e_tests/conf/browserstack.nightwatch.conf.js"
So now I can run my tests just running: $ npm run e2e:browserstack -- --env firefox_win10,safari_catalina
I will get in my console something similar to:

Connecting local
Connected. Now testing…
firefox_win10 Using insecure HTTP connection on port 80. Consider using SSL by setting port to 443 in your Nightwatch configuration.
firefox_win10 [App] Test Suite
firefox_win10 ================
firefox_win10 - Connecting to hub-cloud.browserstack.com on port 80…
firefox_win10
firefox_win10 ℹ Connected to hub-cloud.browserstack.com on port 80 (10779ms).
firefox_win10 Using: firefox (77.0) on windows 10.0 platform.
firefox_win10 Results for: Checking title must contains Hello World! string
firefox_win10 ✔ Element was visible after 1999 milliseconds.
firefox_win10 ✔ Testing if the page title contains 'Hello World!' (418ms)
firefox_win10 ✔ firefox_win10 [App] Checking title must contains Hello World
! string (7.624s)
firefox_win10 Using insecure HTTP connection on port 80. Consider using SSL by setting port to 443 in your Nightwatch configuration.

In my BrowserStack App Live build for Firefox on Windows 10 I will see something like:

So finally, now I just need to add more tests and add it to my CI/CD pipeline.

Aaaaaaaaaand, today we have a super clasic Argentine musician who wrote incredible rock music since 80’s. One of him masterpiece is «El amor después del amor» (it’s Love after love).

 
Ricardo Aragón

Ricardo Aragón

Full Stack Developer iam@ricardoaragon.com