Testing: Choosing a test framework

03 Nov 2022 - Gerrit Grobler

Specifications

I was tasked with finding a way to (potentially) test every single feature our web applications have, whether by Unit Testing or E2E Testing. This is how it’s done:

  • Figure out whether Unit testing or E2E testing is faster and easier to implement, whilst still remaining accurate.
  • Figure out whether code-driven or codeless testing is better in the long run.


Testing Breakdown

Here’s a collection of the testing suites I tried before settling on the best one.

  • Karma - The node-based testrunner packaged with and recommended by Angular, benefits include being: Already installed.
  • Jasmine - The Javascript testing framework you write tests with, benefits include being: Extremely popular.
  • Testim - The super-easy to use AI-powered codeless E2E testing suite, benefits include being: Incredibly fast to make tests with.
  • Cypress - The Javascript testrunner ‘built for humans’ with cross-browser compatibility, benefits include being: Very reliable and simple to code with.


Getting Started

Before I jumped head-first into breaking things and testing how badly I broke them, I had to figure out what the difference between unit testing and E2E testing is, as my experience with testing up until that point was running the code, checking the console logs, and crossing my fingers.


What’s Unit Testing, then?

Well, Unit Testing focuses on testing every single component in isolation, where you use mock services to emulate real calls, which means you can create the perfect environment for testing working code, as well as checking that your error handling functions correctly.

The hardest part here is having the discipline to get your testing mock services set up in the first place, because as soon as you have it, you’ll have it forever. The downside is that any time you add one feature, you need to update the mock service with valid return data as well, in addition to writing the tests themselves.

This is quite the challenge if you’re working on a years-old project with a lot of obscure services.


Alright, E2E testing?

E2E Testing basically has the polar opposite ideology. E2E wants to do it for real, spooling up your frontend and backend, and “physically” interacting with it, as if it was an actual user. It makes real calls to your backend and expects real values to be returned to the frontend.

This means you have to suffer the harshness of reality, as you cannot set up mock services that provide an ideal testing environment, you’ll just test the main environment and if all goes well it won’t break. (There are clever ways around this, like setting up endpoints that purposefully return errors when certain parameters are provided.)

The benefit here is that you don’t have to worry about making the fake services at all, you can just write your test and have it use the thing you just added. This is more so useful in the far-ish future when a different developer or maybe even yourself adds code that breaks previous code.


Trying out Unit Testing

I tried out Jasmine/Karma because our Angular set-up already had the spec.ts files for a vast majority of our components, although most of them were stubs. I spent about a working day or two doing deep research into Unit Testing and writing a test for one of our medium-sized components, as well as creating the mock services for it. Work was slow, extremely slow.

I had gotten it to the point where it was sort of useful but not really because the mock services didn’t really emulate a backend change causing a breaking change, but it was nice to catch compilation errors if they somehow slipped past the developer.

The easiest part was getting the super basics started, as I didn’t need to install any external packages. The hardest part was wrapping my head around the current state of the component, as each test inside the component isn’t really supposed to affect the rest, despite normal application flow being a continuous stream of input until a desired outcome has been achieved.


Hoping E2E Testing Works

Right, my last hope, E2E testing. After doing some research, I discovered that there are a further two methodologies of E2E, one being code-driven and the other being codeless.


Code-driven is when you write code to drive the tests. Here’s a small Cypress excerpt as an example:

it('should navigate to customer page', () => {
  cy.get('#menuButton').click();
  cy.get('#navToCustomers').click();
});

If, for whatever reason, either the get() or click() fails, Cypress will throw an error with should navigate to the customer page being the test that failed.


Codeless testing is when you use a service to record your actions on a web application, which it then replays whenever you run the test. If it can’t find the thing you clicked on, it’ll throw an error.


Code- or codeless E2E?

I tried using Testim, the codeless E2E testing service first, primarily because I spent two days coding tests already and I was a bit tired of it statements. I set up our organization, got a test suite set up, then started recording a simple log-in as an admin, creating a record, verifying that the record exists by opening it, then deleting the record.

It worked absolutely perfectly, I was overjoyed. Then I made a second test, and made a record while doing so. I ran the first test again, and it started deleting the wrong records on my local database. Seemingly, the test only records the CSS selector of the things you click, not necessarily the exact button and word you click, which means the test is extremely easy to accidentally fail or pass.

There are ways around it, by manually putting in verification on each one you click, but what if you miss a step, or something else doesn’t function as you expect and you miss that?


Coding with Cypress

It seems there’s only one option left, so it better work. I installed Cypress on both the front-end (Angular) and the backend (Node.js) and started coding.

util/test/e2e/cypress_test.js

const cypress = require('cypress');

async function runTest() {
	await cypress.run({
    	reporter: 'junit',
    	browser: 'chrome',
    	spec: './angular-src/cypress/e2e/**/*',
    	configFile: './angular-src/cypress.config.ts',
    	config: {
        	baseUrl: 'http://localhost:3000',
    	},
	})
	.then(async result => {
    	await testingCleanup(afterTest = true);
    	if(result.failures) {
        	console.error('Could not execute tests')
        	console.error(result.message)
        	process.exit(result.failures)
    	}

    	// print test results and exit
    	// with the number of failed tests as exit code
    	process.exit(result.totalFailed)
	})
	.catch(async err => {
    	console.error(err.message)
    	process.exit(1)
	})
}

runTest();


angular-src/cypress/e2e/startup/login.cy.ts

describe('Login Test', () => {

  it('should log in successfully', () => {
	cy.visit('http://localhost:3000/login');

	// log in to acquire JWT token
	cy.get('#loginInput').type('cypresstesting+admin@testing.com');
	cy.get('#passwordInput').type('12345678');
	cy.get('#loginButton').click();
  });

})


…and then ran it with node util/test/e2e/cypress_test.js. This worked flawlessly, as I didn’t have the problem with recording having weird selectors I have little control over. I then expanded the test to include opening a menu and selecting the ‘Records’ page:

it('should navigate to customer page', () => {
  cy.get('#menuButton').click();
  cy.get('#navToRecords').click();
});

This also worked, meaning tests could follow a flow very similar to an actual user’s experience of clicking buttons, inputting information, and clicking more buttons.


Going further with Cypress

I decided to look further and see if I could improve my Cypress experience more; I recreated the same scenario where I make a record, verify that it’s there, and then delete it. To do so, I made some custom commands to make the job easier.


commands.ts

function loginAsAdmin(): void {
  cy.visit('http://localhost:3000/login');

  // log in to acquire JWT token
  cy.get('#loginInput').type('cypresstesting+admin@testing.com');
  cy.get('#passwordInput').type('12345678');
  cy.get('#loginButton').click();
}
Cypress.Commands.add("loginAsAdmin", loginAsAdmin);

function getAndTypeInput(inputLabel, inputText?): void {
  cy.get(`input[config-label="${inputLabel}"]`).type(inputText ? inputText : `Demo ${inputLabel}`);
}
Cypress.Commands.add("getAndTypeInput", getAndTypeInput);

function getAndCheckInputVisible(inputLabel): void {
  cy.get(`input[config-label="${inputLabel}"]`).scrollIntoView().should('be.visible');
}
Cypress.Commands.add("getAndCheckInputVisible", getAndCheckInputVisible);


angular-src/cypress/e2e/startup/login.cy.ts

it('should be able to create a new record', () => {
  cy.get('#addRecordButton').click();
  cy.get('#modal-basic-title').contains('New Record');

  // Open a dropdown input and hit the first option
  cy.get('input[config-label="Testing Dropdown"]').click();
  cy.get('mat-option').first().click();

  cy.getAndTypeInput('First Entry');

  // Go to Next Tab
  cy.get('#cdk-step-content-0-0 > form > div > button').click();

  cy.getAndTypeInput('Second Entry');

  cy.get('#saveButton').click();
});

it('should filter for, read, and delete an existing record', () => {
  cy.getAndTypeInput('Filter by First Entry', 'Demo First Entry');
  // Wait for Debounce
  cy.wait(501);
  cy.getAndTypeInput('Filter by Second Entry', 'Demo Second Entry');
  // Wait for Debounce
  cy.wait(501);

  // Click on first AG-Grid row
  cy.get('.ag-center-cols-container .ag-row').first().click();
  cy.get('#modal-basic-title').contains('Edit Record');

  cy.getAndCheckInputVisible('First Entry');

  cy.get('#cdk-step-content-1-0 > form > div > button').click();

  cy.getAndCheckInputVisible('Second Entry');

  cy.get('#deleteButton').click();
  cy.get('#confirmationConfirmButton').click();
});


I added this block after the block that navigates to the ‘Records’ page, maintaining the flow. It clicks the button to open the Add Record modal, confirms that the modal is in the ‘New Record’ state, and then rapidly begins filling in the details, then saves it.

I tried breaking this and seeing what happens, so I removed the ‘First Entry’ field. As you might expect, everything goes well until that point, until it reaches the input it can’t find, it immediately throws an error and stops processing that specific test, then tries to do the next it statement tests, which also fail because the inputs can’t be found.

This isn’t entirely ideal, but it doesn’t cause any problems, so it can just be ignored and the first test that fails can be investigated.


In Summary

For our use-case, where we want to see if a user can navigate our web applications without encountering an error, E2E testing seems to be the best option.

Of all the E2E testing options, I found that Cypress was the easiest to work with and customize.