Visual regression testing for web applications

Tests are important (duh!)

In the software development process, is extremely important to have different kinds of tests to guarantee the correctness and quality of the solution we are building.

“If you don’t like testing your product, most likely your customers won’t like to test it either.” (Anonymous)

There is the "manual" testing, performed by the developers themselves, and then by the QA team. There is also the automated testing (e.g. selenium tests) and unit testing.
All of these test mechanisms serve different purposes and complement each other.


What about tests for CSS?

Noticing CSS and layout problems in a web app is particularly challenging. The most common functional tests usually don't detect these kind of errors. Probably the application is working how it's supposed to work, doing what it should do, just that some margin, font color or button size is wrong.

The details in the layout and visual elements are very important, specially if you have a consumer oriented business. One error in the interface could upset or confuse lots of users.

Usually these layout errors are spotted by the good eye of the UI designer, or some other team member. However, in big applications, with a lot of screens or pages, is impossible to manually test all of the screens every time, to spot all the visual errors that could have been introduced with the latest feature. Hence the need for a way to automate the detection of visual errors.

One way to create automated visual tests, is by validating the css properties, one by one, comparing them to their expected values. However, two different css files could render the same visual output. And this might be exactly the case we need to test.

Imagine you want to rewrite a webpage. You want to build a new, more semantic HTML, with a better, more maintainable CSS, but it has to look the same. The test approach we just described couldn't work for this scenario, because all the test will fail, but the app still could look the way it's supposed to look.

Also, when you have more than one front-end developer, or just many different people making changes on the HTML/CSS, is not uncommon to break some things. New changes in one feature or page can introduce bugs in some other features or pages.

Having good practices and a good CSS architecture, should help us avoid these problems, but anyways, having automated tests is like an extra layer of protection.


My personal experience with visual Regression testing

A couple years ago, I was working for a well known online classified ads website in Chile, and I had to face this challenge. I wanted to rewrite some of the pages, to improve the performance/ load time, improve the semantics of the html, improve the quality of the CSS code, but the product owners required that the website should look the same, so the users won't notice the difference.

Researching a bit about the subject, I found that the most common approach for visual regression testing is capturing screenshots of the app to save as a baseline, and then take new screenshots every time the tests are run, and compare with the baseline to detect differences.

There are many tools to do this. The best one I found at the moment was PhantomCSS

PhantomCSS allow you to write casperjs tests and capture screenshots of different components, or event the whole page.

    start( url ).
        // do something'button#open-dialog');
        // Take a screenshot of the UI component
        phantomcss.screenshot('#the-dialog', 'a screenshot of my dialog');

Creating PhantomCSS tests helped me a lot to detect visual problems as soon as they were introduced to the code. Sometimes these tests even detected functional issues and bugs.

The workflow for PhantomCSS and similar tools is basically the same:

  1. You code the tests.

  2. You run the tests for the first time, saving the baseline screenshots in the process.

  3. You introduce new changes to the code and then run the tests after each commit.

  4. If a test fails, you see the image differences and decide if the test failed due to a legit change or a bug. In the first case you approve the change and update the baseline. If its a bug, you fix the bug.

  5. Run the tests again after solving conflicts.

After working some time with PhantomCSS, I decided to create my own tool, adding some features on top of HaunterJS. That way I built haunterjs

haunterjs, besides having a cool logo, introduces some cool stuff like a web tool to review the screenshots, and help solve the conflicts. Unfortunately, I stopped working on haunterjs, and giving support. That's why it could be a good idea to pick one of the other tools I'm gonna talk about next.


Some other tools

There are some other amazing tools out there for visual regression tests:




Good practices

  1. Visual regression tests are not a replacement for good development practices and a good CSS architecture.
    You still need to create maintainable and scalable code, so you minimise CSS selector conflicts, duplicated code and bugs.

  2. Whatever tool you decide to use, it's a good idea to capture screenshots for small modules or components, instead of capturing the whole page every time. This way the image comparison is faster, and you can update the L&F of a single component, without breaking all the tests.

  3. Integrate your visual tests with some continuous integration platform like jenkins or TravisCI

In summary

Writing test is always important. Visual elements need testing too, specially in a consumer oriented app.

Capturing and comparing screenshots of your website is the way to go. There are some tools like PhantomCSS and wraith to do that.

Include visual tests execution on your CI workflow.

Some other articles I recommend on the subject: