In the dynamic world of software testing, the importance of ensuring the reliability of your end-to-end (E2E) tests cannot be overstated. That’s why we’re thrilled to announce a new experiment allowing you to configure thresholds for test retries!
Context: What’s a retry?
End-to-end (E2E) tests are indispensable for evaluating complex systems, but they can encounter unpredictable conditions that make them unreliable, causing sporadic failures called “test flake”. Flaky tests are those which sometimes fail and sometimes pass when nothing has changed, and therefore give non-deterministic results. These are the notorious tests that manage to slip through your CI and make it into your main branch, initially appearing to pass but later causing havoc as they start to fail intermittently. The only way to determine that a failure is really a failure in your app is to retry the test.
In the past, Cypress's test retries were intended to provide more opportunities for tests to pass, even if they initially failed. However, this approach fell short for critical workflows within applications, where a low tolerance for risk demands immediate notification of failures.
What’s new with experimental retries
You can now specify what level of test flakiness should pass or fail by setting fine-tuned thresholds for retries based on what’s important to you:
- You can detect flake but pass on a threshold that you configure, or
- You can detect flake but always fail to ensure your team is notified
This extra configurability allows you to:
- Reduce noise to keep you team’s attention focused on your most important work
- Increase agility so you can respond quickly to issues if you want to treat flaky tests in your most critical workflows with the same urgency as failures
- Increase control over CI resources and time
How to get started
Simply set the experimental flags to true in your Cypress config to get started. Visit our documentation for detailed instructions on using this feature. Keep in mind that at launch, this feature can only be configured at the global level.
What’s next
This experiment is part of a larger project at Cypress to enhance your workflow for detecting, understanding, and resolving test flake. We are building the most advanced flake detection feature set on the market for E2E testing so you can catch flake as soon as the root cause is introduced, before it gets into production. Stay tuned to our Twitter, blog, or Discord for updates.