Use meaningful smoke tests

Let's say you cover your application with end-to-end tests. You achieve the elusive 100% code coverage using the Cypress code coverage plugin. Is this enough to guarantee a good user experience?

Maybe, but I'm not so sure.

Another question to consider is: as Cypress.io prepares to release its cross-browser support with Firefox, would you run ALL your tests in Electron, then again in Chrome, and then once more in the Firefox browser? Is running all functional tests in an evergreen browser likely to reveal any bugs - 3 times as likely as running all your tests in bundled Electron browser for example?

I think not.

Instead, we propose running all tests in one browser, and running just the smoke tests in other browsers. If there is a potential problem that completely stops the Firefox users from using your web app - it probably will fail the smoke tests.

Let's uses the TodoMVC application in cypress-example-todomvc-redux as a concrete example. Its code is completely covered by a combination of E2E and unit tests. Now let's write a single smoke test that verifies just the most important features of the Todo application: adding items, completing items, viewing completed and remaining todos. Here is a smoke test function from the cypress/integration/smoke.js file.

/**
 * This test goes through a longer user story
 * trying to do almost everything a typical user would do.
 */
export const smokeTest = () => {
  cy.visit('/')
  cy.log('add 3 todos')
  cy.get('.new-todo')
    .type('write code{enter}')
    .type('write tests{enter}')
    .type('deploy{enter}')
  cy.get('.todo').should('have.length', 3)

  cy.log('1st todo has been done')
  cy.get('.todo')
    .first()
    .find('.toggle')
    .check()
  cy.get('.todo')
    .first()
    .should('have.class', 'completed')

  cy.log('by default "All" filter is active')
  cy.contains('.filters a.selected', 'All').should('be.visible')
  cy.contains('.filters a', 'Active')
    .click()
    .should('have.class', 'selected')
    .should('be.visible')
  cy.get('.todo').should('have.length', 2)

  cy.log('check "Completed" todos')
  cy.contains('.filters a', 'Completed')
    .click()
    .should('have.class', 'selected')
    .should('be.visible')
  cy.get('.todo').should('have.length', 1)

  cy.log('remove completed todos')
  cy.get('.clear-completed').click()
  cy.get('.todo').should('have.length', 0)
  cy.contains('.filters a', 'All')
    .click()
    .should('have.class', 'selected')
    .should('be.visible')
  cy.get('.todo').should('have.length', 2)
}

You can run just the smoke test by itself via smoke-spec.js test file.

Note a couple of things. First, I prefer using cy.log to leaving comments, because log messages are visible in the test run video and screenshots.

Second, the smoke function is exported from the smoke.js file - it is NOT a test (spec) file. Instead we can use the smoke function from other tests. For example, we can run the smokeTest function from the smoke-spec.js file.

import { smokeTest } from './smoke'

it('does not smoke', () => {
  smokeTest()
})

The code coverage collected from smoke-spec.js tells us we are testing a majority of the application's code - 84% of it.

We can drill down to the reducer file to see the application features the smoke test does not cover - very few it turns out.

We could extend the smoke test to quickly cover those features, if we deem them important enough. Even better, we can cover different window resolutions rather than the code lines. Will our application work on the desktop AND on the mobile screen? Here is the cypress/integration/viewports-spec.js file.

import { smokeTest } from './smoke'

Cypress._.each(['macbook-15', 'iphone-6'], viewport => {
  it(`works on ${viewport}`, () => {
    cy.viewport(viewport)
    smokeTest()
  })
})

By running the viewports_spec.js we confirm that most application features are working on both screens - no buttons or links are occluded, the Test Runner can click and access everything the user needs to create items, mark them completed, etc.

Finally, every time we deploy the application, we can quickly run just the smoke test confirming the deploy has succeeded. We can pass the url to test and the spec filename via command line:

npx cypress run --config baseUrl=http://localhost:1234 \
  --spec cypress/integration/smoke-spec.js

If you have more than a single Cypress configuration parameter to use a custom configuration file option `--config-file`. Place all smoke options into their own JSON file, like cypress-smoke.json to be used instead of cypress.json. Then run the smoke test whenever needed with:

npx cypress run --config-file cypress-smoke.json

Finally, when we have Firefox support, we can run the full set of tests in Electron, then just the smoke test in FF.

npx cypress run
npx cypress run --config-file cypress-smoke.json \
  --browser firefox

Note: you can find the source code for this post in cypress-example-todomvc-redux