Customers Optimizely

Optimizely is dedicated to enabling companies to experiment with and swiftly release software, empowering them to test and learn in production before rolling out new features universally. The platform leverages feature flags and progressive rollouts to simplify A/B/n testing, personalization, multivariate testing, and stats acceleration. Feature flags, a fundamental aspect of the platform, enable runtime control over functionality without deploying new code, facilitating experimentation throughout the entire feature lifecycle.

Challenge

Founded in 2018, the Optimizely QA team was tasked with assessing the quality of the platform and devising strategies for enhancement. Prior to the formation of the QA team, various engineers within the organization contributed to a customized Selenium-based end-to-end testing framework established by a former employee. However, when this employee departed, the framework lacked test cases and centralized maintenance. The framework was plagued by flaky tests, unclear feature coverage, and significant difficulty in usability and debugging. Regression bugs necessitated frequent hotfixes post-production release, while developer resistance to Python and sluggish pipeline performance compounded the challenges. The QA team’s assessment revealed inadequate regression test coverage, with only 72% coverage in the Experimentation Platform and zero automation coverage in non-experimentation areas.

Solution

Responding to the pressing need for a more effective testing solution, the QA team explored Cypress, drawn to its promising features and positive industry reputation. Implementing a proof of concept, they utilized Cypress to address the regression test coverage gaps, concurrently evaluating its performance against the incumbent platform. The trial proved successful, with engineers significantly enhancing regression test coverage within a month. Cypress’s benefits included its developer-friendly learning curve, JavaScript-based platform, fast test creation, streamlined debugging of flaky tests, and simple test maintenance and organization.

Result

Over a single summer, just two QA engineers leveraged Cypress to augment regression testing of the Optimizely Experimentation Platform by 28%, achieving full feature coverage and substantially mitigating incoming highseverity issues. Integration with Cypress Cloud yielded a remarkable 300% improvement in test performance, with tests now running across 16 machines for two products and one third-party integration. Cypress’s intuitive features, such as time travel capability and automatic failure screenshots, have streamlined debugging processes for the QA team. With the legacy Selenium framework sunsetted, the entire end-to-end test suite is transitioning to Cypress, poised to enhance efficiency tenfold. Front-end engineers are now proficient in writing end-to-end tests, empowered by Cypress’s user-friendly functionalities and robust capabilities.

We made the decision to sunset our legacy Selenium automation framework, and there’s no doubt it was the right one. Soon, our entire end-to-end test suite will be moving to Cypress, and our efficiency will go up tenfold.