Customers Gatsby

Gatsby is a free, open source site generator that helps developers build blazing fast websites and apps. With the framework’s emphasis on on performance and accessibility, testing everything quickly and thoroughly is crucial. Find out how end-to-end (E2E) testing with Cypress has helped the Gatsby team accept more open source contributions than ever before.

A rapidly-growing community

Gatsby is an incredibly fast modern site generator that weaves together the best parts of React, Webpack, internal routing, GraphQL, and other front-end tools into a seamless, developer-friendly experience.

Since launching in July 2017, almost 2,500 contributors and tens of thousands of developers* build personal and commercial websites with the framework. With upwards of 600 pull requests (PRs) submitted every month—and the majority (75%) submitted by active community members—it’s crucial for the team to have a robust method to quickly test these PRs.

*As of September 2019.

Before: Contributions hampered by bugs

Feedback from a contributor was a wake-up call: “[One user] said Gatsby is really hard to contribute to,” says Gatsby Software Engineer, Dustin Schau. “He was surprised by how many interrelated bugs were in the code, and had to spend a few days fixing them. Even then, he wasn’t sure everything was working as it should be.”

The main problems were:

• Contributors and maintainers were battling to merge PRs confidently, unless they went through a manual, error prone process.

• Contributors were worried they might break something unrelated.

• Failed status checks gave no clear insights for fixes.

• PR review times were very long.

The Gatsby team was fixing all these bugs reactively, manually and with long review processes. They needed a way to catch bugs before shipping to end users, with more certainty and confidence.

Cypress has been able to increase our confidence and certainty with pull requests. We know we’re getting it right proactively, rather than after the fact with retroactive, follow-up PRs.

“Crazy good developer experience”

Despite a simple Travis test suite, some Jest unit tests, and rudimentary UI-type E2E tests, code was still failing – often for strange reasons.

The team trialed several tools and libraries including Puppeteer, Selenium and Nightwatch. All had value, but none were quite right for Gatsby’s specific needs. They ultimately chose Cypress because:

• Cypress combined with CircleCI delivers very reliable CI runs.

• It’s JavaScript-based, and very easy to understand and get started.

• Cypress is closer to the browser, enabling BDD testing in an environment that matches how Gatsby is used closely.

• Cypress includes useful helpers and utilities, and has an intuitive and clean syntax.

• Cypress’s developer experience includes instant feedback and insights as to exactly where things went wrong.

• The NPM trends graph showed that Cypress is growing faster than other tools, indicating healthy, strong support.

After: Confidence and E2E priority

With Cypress, Gatsby has created a robust suite of E2E tests - a rock solid foundation they continue to improve upon. The tests help everyone (particularly new team members) understand how the code is expected to work.

Seeing why tests failed and the unexpected impacts elsewhere has given the team greater confidence. The team also has a much more extensive view of how end users are using Gatsby.

The team’s workflow has changed, with E2E tests becoming the most valuable part of their code base. Both Gatsby’s developers and community contributors are using Cypress’s simple commands to write E2E tests for their PRs – and managing repetitive tasks with custom commands.

We’re catching even the weird regressions at PR time, instead of afterwards. We know we’re not going to break anything in unforeseen ways.

Because we now have a trusted E2E suite, when someone new joins Gatsby or becomes a community contributor, their first PRs are quicker and smoother, as they know they’re not breaking something fundamental.

Future plans

Gatsby is all about speed, and the team is discovering how to make their tests even faster. They run a suite of around 20 CI checks for each PR, so the next step is to investigate running Cypress tests in parallel. As Gatsby adds new product features and gathers more contributions, the team will continue to expand their E2E test suite—and their reinvention of modern web development—with Cypress.