cypress e2e solution

cypress.io

• alternative to phantomJs, Karma, Selenium, ...
• in private beta (still in 2017)
• not already opensource but coming
• it's done :-)

features and objectives

• fast (running as fast as unit testing)
• reliable (no inconsistent errors like in Selenium) (inconsistents errors examples in Fixing the flake on CI : http://samsaccone.com)
• rewindable (you can see what the test was doing step by step)

trade offs

Cypress Trade Offs

environment and browser support

Proposal: Support for Cross Browser Testing

we only support Chrome variants such as Chrome, Chromium, and Canary.

We will be adding cross browser support likely by EOY. (20171017) Firefox, Safari then Edge.

native browsers events support

Support for Native Browser Events

From there we'll add new commands for file uploads, etc.

installing with npm behind a proxy

• npm install not working behind corporate firewall
• Document CYPRESS_BINARY_VERSION environment variable

comparison Cypress vs TestCafe

Evaluating Cypress and TestCafe for end to end testing - medium.com/yld-engineering-blog - 20180329

Compare to Cypress.io - testcafe-discuss.devexpress.com - 20171010

with post by TestCafe dev

usage and recommandations

Cypress slows down drastically over long runs - restart renderer between spec files?

You cannot run all of your tests. The GUI is a mode built specifically to run one or a small number of tests at a time.

you can tell Cypress to cleanup snapshots by setting the { numTestsKeptInMemory: 0 } in cypress.json. This will purge snapshots aggressively but then you will lose the ability to debug any of your tests.

If you want to run all your tests you need to do it from the command line using cypress run. In that mode it makes specific performance optimizations (like not taking snapshots) because it knows you're not iterating on a test or working with the results.

Tests slows down to a crawl and eventually times out/crashes

The reason Cypress is crashing and chrome headless isn't is because of the additional features Cypress provides that no other testing tool does. In this case it's the time travel functionality.

Cypress has two basic modes of operation: GUI mode & CLI mode

GUI mode is optimized for debugging and CLI mode is optimized for running.

The reason Cypress is slowing down and crashing is because in GUI mode it takes snapshots for very single command.

window and document are being preserved in memory for every single page transition. This is where it chews up memory and you'll experience those slow downs and crashes.

My test passes cypress open but fails with cypress run

One is running in Chrome, the other is running in Chromium (Electron). There are differences in these browsers.

Cf documentation - Launching Browsers : Electron Browser

doc off best practices - selectors (need cypress 2.1+)

Use data-* attributes to provide context to your selectors and insulate them from CSS or JS changes.

login

Best Practices - Visiting External Sites

Web Security - External Navigation

Web Security - Form Submission Redirects

Cypress.io Keycloak Integration : 20171028

how a test works

selenium

Between each action we wait a defined amount of time to be sure the browsers have rendered the DOM needed for the test.

cypress

Cypress runs inside the browser.

The test suite execute the script and queue all the commands. Then cypress run everything in the same order. Cypress execute the first command and retry until success or fail (defined timeout). This modus operandi is way faster then selenium.

Each command takes a snapshot of the current DOM.

Numerous commands contains their own embedded assertions.

See core concepts documentation > Chains of Commands > Commands Are Asynchronous.

articles

Web testing nirvana with Cypress

Installing and Running

Control an AngularJS Application From E2E Tests : 20171115

slides

Testing Vue Apps with Cypress.io (STLJS Meetup April 2018) - Christian Catalan - 20180419

awesome context, excellent slides

Automated Testing for the Modern Web - Jennifer Jeshane - 20180509

good e2e history prez

videos

Brian Mann, Testing the way it should be : ReactiveConf 2016

Brian Mann creator of cypress.io

cypress is a testing tool running entirely in javascript

Why another testing tool ?

Sharing is own experience in the first part :

• 9 years developping webapp including 5 with testing
• testing is too hard
• tools are too old (selenium is born in 2004, cf Selenium history)
• needs a dedicated Q/A team to works well
• Brian talks to a lot of developpers to know what they need in remplacement of actual tools

What developpers needs ?

3 recurring things developpers finds challenging :

• Setup (sane config, less hassle to set up, etc ...)
• Writing (Simpler to write, testing responsive, see visually what broke, async, etc ...)
• Management (clear and direct way to understand WHY a test have failed)

Selenium (or webdriver) is the core tool. Embedded in Protractor, Nightwatch, ...

At that time, all the webapp were simple and staless (full page refresh at each user action). Javascript was not to build webapp but to decorate existing webapp. Most of the code was synchronous. Nowadays, there is SPA, stateful, no page refresh and everything is asynchronous.

Selenium original flaw is : it is a stateless http API. It is impossible to test effectively a stateful app with a stateless API. Selenium cannot know and show a particular state at a particular moment.

This forces dev to build tests (e2e) after everything is built (impossible to TDD)

How cypress solves the problem ?

Part 1: Setup

npm install -g cypress-cli && npm install cypress

cypress open

You are done. You can write your first test.

Part 2: Writing

Working on reporting, when a assertion pass, and when it fail. Lot's of information for the developer.

Snapshoting : Every command ran by cypress takes a snapshot of the DOM. You can open your dev tools and check the logs pushed by cypress with the state of the app.

Events : They are captured during execution an logged into the console.

Frameworks : Cypress is built on top of Mocha, Chai and Sinon.

Async : Mostly hidden, each command return promise under the hood.

Hooks : onBeforeLoad hook for cy.visit command allows developer to : open the browser debugger, alter the global state of the app (for example by injecting user info or tokens). You can stub server responses (very nice to tests error messages and related ui behavior)

Part 3: Management

CI related. When a test suite will be ran in a CI provider (TravisCI for ex), output will be sent to cypress, then cypress will sent it back to cypress desktop app.

Can output logs, screenshots, even video of the suite

Ability to remote connect to the CI to see and manage the suite run.

Goals for cypress

• Open Source
• Champions
• Documentation
• Happiness

Roadmap

They are only 5. They have raised some money. Not all repo are opensource now.

Questions

Not supporting IE right now, it will. (Brian thinks crossbrowser testing is going down because there is less and less difference between browsers).

Jack Franklin, Testing React Applications : React Amsterdam 2016