No one likes flaky tests, not even their mama. In this blog post I will show a test that appears simple, yet acts in surprisingly annoying and frustrating ways. We will slightly change the test to make it reliable and flake-free. We will apply the following three principles to the test refactoring to achieve this:
- every assertion should pass for the right reason and not accidentally
- the page navigation requires assertions around it to avoid the dreaded element detached from the DOM error
- the test runner should ensure the application is ready to receive user events before sending them
The source code for this post can be found in the repository next-js-example-may-2020, based on the reproduction  example from the many comments on the issue #7306.
The Test
Let's look at the original test. The test loads the home page /, clicks on the "About" link, which takes the browser to the /about page. On that page the test asserts the text "About" is present, then navigates to the /users page.
context('Navigation', () => {
it('can navigate around the website', () => {
cy.visit('http://localhost:3000');
cy.get('[data-cy="header-link-about"]').click();
cy.get('main:contains("About")');
cy.get('[data-cy="header-link-users"]').click();
cy.get('main h1:contains("Users")');
});
});The test passes (most of the time)
The Red Flags
Looking at the Command Log raises red flags, despite the test passing. I have marked the two suspicious items in the screenshot below.
- the test has interacted with the navigation link "About" but shows these elements as no longer visible (the crossed eye icon). Cypress has visibility checks built into the cy.click command, so the element should stay visible. Has something changed during or after the click?
- the last command of the test checks if the text "Users" is present on the page inside the
<H1>element - but the element is found before the web application loads the/userspage. We expect the text to be found after that page loads!
Let's look at the test with a careful eye to make sure a passing test does not mask any underlying problems.
Do The Right Thing
Every assertion in our test should pass for the right reason. We want to confirm the text "About" is shown when we navigate to the /about page. We want to confirm the text "Users" is shown when we navigate to the /users page. Let's look at our test steps - by hovering over the commands we can see which element the assertion used during the test execution.
The DOM snapshot and the element highlight shows the test command cy.get('main:contains("About")'); passed accidentally. It matched the "About" text on the / page, not on the /about page.
I love using the Cypress time-traveling debugger to inspect the test commands one by one to make sure they are finding the right elements and pass for the right reason and not accidentally. Let's change the test - we really want to confirm we are on the right page first, then confirm the text contents. Here is the updated start of the test.
it('can navigate around the website (better)', () => {
cy.visit('http://localhost:3000');
cy.get('[data-cy="header-link-about"]').click();
cy.location('pathname').should('match', /\/about$/);
cy.contains('main h1', 'About').should('be.visible');
});Tip: I recommend using the cy.contains command to find elements by text or by regular expression.
The above test ensures we are on the /about page before searching for the <H1> element containing the text "About". Similarly we should update the "Users" check.
Detached Element
Before we continue let's take a look at another flaky error this test occasionally exhibits. This error is more likely to manifest itself on the continuous integration server, but we can hit it locally by running the test multiple times in a row.
I have written about this type of error in a separate blog post Do Not Get Too Detached. In this instance the Command Log again tells us what happens.
- The application loads the index page
/ - The test runner clicks on the "About" navigation link
- The test runner then finds the "About" text (accidentally as we saw in the previous section)
- The test runner then finds the "Users" navigation link
- The application meanwhile reacts to the "About" navigation event and loads the page
/about, completely replacing the entire DOM tree with the "About" page - The test runner still has the reference to the "Users" navigation
<a>element - but that element is no longer in the current DOM on the page. - Cypress throws an error, failing the test
The Test Runner and the application are out of sync - they are racing against each other, creating unpredictable outcomes. We need to "synchronize" the Test Runner - it has to make sure the application has finished updating in response to a test command before executing the next test command. We can achieve this by adding assertions after the cy.click command - we must wait for the application to load the new page before we can run any more test commands.
Luckily, what we have done in the previous section of this blog post is enough, because we have used the cy.location('pathname')... assertions to ensure the Test Runner "waits" for the new page to load.
it('can navigate around the website (better)', () => {
cy.visit('http://localhost:3000');
cy.get('[data-cy="header-link-about"]').click();
cy.location('pathname').should('match', /\/about$/);
cy.contains('main h1', 'About').should('be.visible');
cy.get('[data-cy="header-link-users"]').click();
cy.location('pathname').should('match', /\/users$/);
cy.contains('main h1', 'Users').should('be.visible');
});The above test alternates test commands with assertions ensuring the application has finished its updates and is ready to receive the next test command.
Visible Elements
Let's look at the last red flag the Command Log shows at the start of the test - the invisible navigation link.
The Cypress Test Runner should not be clicking an invisible element - because the user cannot click it, and Cypress tries to act like a human user would. Thus something is happening after the click, making the original clicked element no longer visible. We can see that the page's elements are invisible when we print them - because the DOM snapshot shows no navigation links when we look back at the page state.
Let's observe our web application while it loads. The DevTools console shows the DOM elements changing with additional attributes being added.
If the <body> element is being hydrated, that probably means the individual elements on the page, like navigation links, are being hydrated and probably change. The Test Runner should be aware of this process and let the application fully hydrate before interacting with the elements. In the simplest terms, this could mean confirming the navigation link is visible before clicking on it.
cy.get('[data-cy="header-link-about"]').should('be.visible')
.click();The above command ensures the element has finished hydrating before clicking it and again interleaves the test commands and assertions to ensure the application is ready to receive the next command.
it('can navigate around the website (final)', () => {
cy.visit('http://localhost:3000');
cy.get('[data-cy="header-link-about"]').should('be.visible')
.click();
cy.location('pathname').should('match', /\/about$/)
cy.contains('main h1', 'About').should('be.visible');
cy.get('[data-cy="header-link-users"]').click();
cy.location('pathname').should('match', /\/users$/)
cy.contains('main h1', 'Users').should('be.visible');
});The full test runs and shows the accurate DOM snapshots for every navigation command.
Notice the full test interleaves commands and assertions to make sure the test and the app are not blindly racing each other.
Conclusions
- Even if the test is passing, inspect it by hovering over every command to make sure every command is working with the intended element and does not pass accidentally.
- Alternate test commands and assertions to ensure the Test Runner stays "in-sync" with the application under test and does not blindly push the buttons while the application is struggling to keep up.