Customers Prodigy EMS

Recognizing a significant gap in e-Learning resources, the team behind Prodigy EMS embarked on a mission to develop an online continuing education platform that would exceed industry standards. Following numerous internal iterations, Prodigy was launched in late 2016 with the goal of seamlessly guiding users through the recertification process, complete with state and department-specific training modules. EMS Training Officers are granted comprehensive access to review class records, ensuring compliance with certification standards and eliminating surprises.

Shortly after its launch, Prodigy experienced rapid expansion of its codebase alongside the increasing complexity of deployments. In response to this growth, the team recognized the need for a robust testing solution to swiftly address bugs while maintaining delivery speed and upholding product quality.

Challenge

The team was initially reluctant towards testing, as they perceived it as a time-consuming task for their very small and scrappy team. However, their skepticism faded as their releases were often interrupted by major bugs, disrupting Prodigy’s commitment to quality and delivery. The team needed a tool that would not only enable them to catch bugs before production but also be efficient and user-friendly.

I always had mixed feelings about testing, particularly within small teams, as it often consumes a significant amount of time.

My experiences have been varied; while I’ve witnessed successful implementations, more often than not, testing has been inefficient. There’s a tendency for excessive coverage of trivial aspects and insufficient coverage of complex scenarios due to their challenging nature to test.

This negative perception persisted, exacerbated by the frequent occurrence of major issues slipping through our releases without detection. We faced the dilemma of whether to invest in human QA resources or explore outsourcing options. It was during this period, coinciding with the rise of front-end test automation, that Cypress emerged as a prominent choice.

Solution

After transitioning to the Cypress Cloud Team plan, Prodigy’s testing strategy underwent a significant transformation. With Cypress, they immediately ceased experiencing costly oversights. A critical instance occurred early on when their testing uncovered a race condition flaw in their login screen just before release—a potential catastrophe averted. As their test suite matured, the team’s confidence in rapid releases grew. This accelerated pace, coupled with smaller change sets, not only enhanced their efficiency but also bolstered the overall quality of their deliverables.

As Prodigy’s team expanded and their projects became more intricate, they quickly realized that their existing monthly pricing plan with Cypress couldn’t keep pace. With the addition of another engineer and the continuous development of complex features, Prodigy’s test suites grew exponentially. Instead of removing tests, they found ourselves consistently adding to them, resulting in a compounding effect on their testing volume with each run. Transitioning to an annual Business plan provided a more cost-effective solution, alleviating the burden of increasing testing demands.

However, what truly proved invaluable was the unexpected benefit of early termination of test suites upon failure, via Cypress Cloud’s Auto Cancellation feature. Initially, Ethan and team hadn’t anticipated its significance, but its implementation has significantly streamlined their testing processes, allowing them to identify and address issues swiftly.

The cost of running our end-to-end test suite multiple times a day can quickly escalate. Automatic cancellation upon detecting obvious errors has proven to be a valuable cost-saving measure for us, significantly mitigating the expenses associated with extensive test runs. This isn’t something we would’ve had access to on our Team plan.

Implementing Spec Prioritization alongside Auto Cancellation also yielded unexpected insights beyond its immediate advantages. While it enabled the Prodigy team to promptly address failed specs and troubleshoot issues efficiently, they discovered an intriguing consequence. It altered the sequence of our test runs, revealing instances where their tests were structured incorrectly and dependent on one another.

That’s huge because tests that are dependent on one another are more fragile. Changes to one test or its environment can cause cascading failures in other tests. Independent tests allow my team to identify and address specific problems without interference from other parts of the codebase. This ultimately enhances the accuracy of debugging and troubleshooting processes.

Implementing effective testing practices is also crucial for Prodigy’s development team, especially given their reliance on third-party integrations for critical functionalities like video uploads. In the past, diagnosing and resolving test failures was a time-consuming process, often requiring manual intervention to understand the root cause. As a result, the launch of Test Replay proved to be an impactful addition to their Cypress Cloud suite.

Previously, when encountering test failures, we relied on capturing screenshots and logs in CircleCI, which often lacked context and required tedious manual efforts to reproduce failures locally. With Test Replay, particularly beneficial for flaky tests, we now have the ability to quickly visualize and understand test failures. This has been instrumental in addressing issues caused by unreliable third-party SDKs, such as failed video uploads, by providing detailed insights into the failure scenarios. The Test Replay feature has significantly enhanced our debugging process, allowing us to identify and resolve issues more efficiently than ever before.

Result

Since subscribing to Cypress Cloud, the frequency of Prodigy EMS’ releases has significantly increased 4X over time. Previously, they lacked confidence in their release process, scheduling them cautiously and allowing ample time for potential fixes. Now, with improved testing practices and confidence in their automation tools, they’re comfortable with more frequent releases, even on Fridays or weekends.

This accelerated release cadence has resulted in faster delivery of new features and quicker resolution of bugs, as all updates are bundled into the same codebase. My developers also express greater satisfaction as they witness their work going live more swiftly, maintaining momentum and avoiding prolonged waiting periods for testing.

Furthermore, Ethan’s role has undergone a significant transformation from primarily implementing functionality to predominantly reviewing it. This shift has had a profound impact on his workflow. Now, when developers hand off completed features, he can rely heavily on an accompanying test suite integrated into the codebase.

Previously, I would have depended on documentation and instructions, but now, end-to-end tests serve as comprehensive guides. This change in workflow not only streamlines our processes but also ensures consistency and longevity in our testing practices.