BlogEducation: Tutorials

When Can The Test Stop?

January 16, 2020

•

By Gleb Bahmutov

Imagine a small web page application that shows a window Confirm popup when the user clicks a button. The HTML might look like this

<html lang="en">
  <body>
    <main>
      <h1>Assertion counting</h1>
      <p>Click on the button, should show Window confirm dialog</p>
      <button id="click">Click</button>
    </main>
    <script>
      const confirmIt = () => {
        window.confirm('Are you sure?')
      }
      document.getElementById('click').addEventListener('click', confirmIt)
    </script>
  </body>
</html>

The application works.

Window Confirm application

We can write a test to confirm (pun intended) that the application works:

/// <reference types="cypress" />
describe('Window confirm', () => {
  it('calls window confirm', () => {
    cy.visit('index.html')
    cy.on('window:confirm', (message) => {
      expect(message).to.equal('Are you sure?')
    })

    cy.get('#click').click()
  })
})

Cypress shows the test is passing.

Passing Cypress test

Note: you can find the source code for this blog post in the Blogs section of the Cypress Example Recipes repository.

But there is a subtle problem with this test, and it becomes obvious when the application under test becomes a little more realistic. Imagine the application shows the Confirm prompt after a delay - maybe the web application needs to make an asynchronous call to the server before showing the prompt. Let's change the source code and add a 1 second delay between the button click and the window.confirm call.

const confirmIt = () => {
  // show the confirm prompt after a delay
  setTimeout(() => {
    window.confirm('Are you sure?')
  }, 1000)
}
document.getElementById('click').addEventListener('click', confirmIt)

Our test is still passing.

Confirm the prompt test is passing even with delayed call

Notice how the test finished after 140ms, yet the assertion gets added after 1 second. Let's see what happens if the assertion fails.

/// <reference types="cypress" />
describe('Window confirm', () => {
  it('calls window confirm', () => {
    cy.visit('index.html')
    cy.on('window:confirm', (message) => {
      // make the assertion fail on purpose
      expect(message).to.equal('A test')
    })

    cy.get('#click').click()
  })
})

The test ... still passes, even with a red failing assertion shown:

The test is not failing as expected

We have a problem - our test finishes before the assertion runs. Thus, the assertion when failing cannot change the status of the test. Even worse - if there are tests that follow, the failing assertion can break those tests instead.

A test should not stop until all assertions have passed. We can modify our test to "wait" for the expect to execute, and we can do it in a variety of ways.

it('waits for window confirm to happen using spy', () => {
  cy.visit('index.html')
  cy.on('window:confirm', cy.stub().as('confirm'))

  cy.get('#click').click()
  // test automatically waits for the stub
  cy.get('@confirm').should('have.been.calledWith', 'Are you sure?')
})
The test does not finish until the stub has been called

We can fail the assertion on purpose - the test behaves as expected

it('waits for window confirm to happen using spy', () => {
  cy.visit('index.html')
  cy.on('window:confirm', cy.stub().as('confirm'))

  cy.get('#click').click()
  // test automatically waits for the stub
  // make the assertion fail on purpose
  cy.get('@confirm').should('have.been.calledWith', 'A test')
})
The failing assertion fails the test as expected

We can also ensure the window:confirm event has happened by setting a local variable and re-trying should(cb) until it has been set.

it('waits for window confirm to happen using variable', () => {
  cy.visit('index.html')
  let called

  cy.on('window:confirm', (message) => {
    expect(message).to.equal('Are you sure?')
    called = true
  })

  cy.get('#click').click()
  // test automatically waits for the variable "called"
  cy.wrap(null).should(() => {
    expect(called).to.be.true
  })
})

The test waits automatically and does not finish prematurely.

Waiting for the variable to be set using should retries

We can also chain a Promise-returning function to Cypress commands - the test again will wait for the promise to resolve before finishing

it('waits for window confirm to happen using promises', () => {
  cy.visit('index.html')
  let calledPromise = new Promise((resolve) => {
    cy.on('window:confirm', (message) => {
      expect(message).to.equal('Are you sure?')
      resolve()
    })
  })

  // test automatically waits for the promise
  cy.get('#click').click().then(() => calledPromise)
})
The test waits for the linked promise

We can even use a very popular plugin, cypress-wait-until, to wait for a predicate:

import 'cypress-wait-until'
it('waits for window confirm to happen using cy.waitUntil', () => {
  cy.visit('index.html')
  let called

  cy.on('window:confirm', (message) => {
    expect(message).to.equal('Are you sure?')
    called = true
  })

  cy.get('#click').click()
  // see https://github.com/NoriSte/cypress-wait-until
  cy.waitUntil(() => called)
})
Waiting for a predicate using cypress-wait-until

Assertion counting

In all of the above tests, the user had to write code to ensure the tests finish only after all assertions have run. We can generalize this feature and implement a number of expected assertions in the test. The user should declare how many assertions the test has, and the Test Runner can check after each test if the number of actual assertions counted matches the expected number. If the number is different, the test fails. This feature is present in some testing frameworks, see t.plan in Ava and Tape. I have implemented a similar feature as a user plugin cypress-expect-n-assertions. Here is our test again - I will make it fail on purpose (otherwise the check is silent):

import { plan } from 'cypress-expect-n-assertions'
it('waits for window confirm to happen using stub', () => {
  plan(0) // set wrong number of expected assertions
  cy.visit('index.html')
  let called

  cy.on('window:confirm', (message) => {
    expect(message).to.equal('Are you sure?')
    called = true
  })

  cy.get('#click').click()
  // see https://github.com/NoriSte/cypress-wait-until
  cy.waitUntil(() => called)
})

The test finishes after 1 assertion has run, but because of plan(0) it fails for this demo:

Counted vs planned number of assertions check failing on purpose

The plugin cypress-expect-n-assertions has one more surprise under its sleeve. If the number of counted assertions during the test is lower than the expected number, it waits automatically for more assertions to run (up until defaultCommandTimeout). Thus our test can simply be:

import { plan } from 'cypress-expect-n-assertions'
it('waits for planned number of assertion to run', () => {
  plan(1)
  cy.visit('index.html')

  cy.on('window:confirm', (message) => {
    expect(message).to.equal('Are you sure?')
  })

  cy.get('#click').click()
})
Automatic waiting for the expected number of assertions to arrive

That's pretty slick.

See also