Visual assertions, also known as visual regression tests, are used to detect changes in your UI across test executions via screenshots. In other words, Autify Nexus can tell whether a certain part of your website looks different than it did at the time of recording or the last time a test ran successfully.
Recording a visual assertion step
To record a visual assertion, choose one of the following actions when recording:
Assert page visual change (page assertion)
Looks the same (element assertion)
Max pixel difference (%)
The max pixel difference sets a threshold of acceptable change, literally the difference in pixels, of the target between test executions. Changes beyond the threshold will cause the assertion to fail. 0% is very strict — a pixel perfect test. 100% accepts all changes to the target.
Setting the correct threshold for failing assertions
Depending on your computer’s hardware and software settings, as well as the window size you use during test execution, your visual assertion may not pass on the first test you run after recording. To keep things simple, you can use the screenshot recommended by Autify Nexus by clicking Update expected screenshot in the detailed test result for the failed test. If you use this screenshot, you do not need to adjust the threshold of your assertion. Subsequent tests will pass as long as there is no further change to the target.
If you expect minor changes in your UI, minor rendering differences across test environments, or you want to stabilize an assertion against a complex element, try increasing the threshold in 1 to 5% increments until you find a reasonable level that works for you.
FAQ
How should I choose between the Looks the same and Assert page visual changes assertions?
The Looks the same assertion (Element assertion) verifies that the appearance of a specific designated element has not changed. On the other hand, the Assert page visual changes assertion (Page assertion) verifies that the appearance of the entire page has not changed.
Guidelines for choosing between them are as follows:
Cases where Looks the same is appropriate
When you want to verify if the appearance of a specific element on the page is as expected.
When testing a dynamic page where the content changes upon each access, and you only want to verify a specific unchanging element.
Cases where Assert page visual changes is appropriate
When you want to verify if the layout and display of the entire page are as expected.
Why do tests fail with minor differences in image comparison (visual assertion) steps involving text, depending on the execution environment such as local or cloud environments?
Depending on the execution environment, minor differences occur in image comparisons involving text because the font drawing and rendering methods differ across operating systems. Please consider the following methods:
Change to text assertions
If the primary purpose is to verify the consistency of text information, we recommend changing to an assertion that verifies the text content itself, rather than using image comparison.
Adjust the threshold
You can adjust the comparison threshold by increasing it to allow for minor differences.
Split scenarios by execution environment
It is also effective to split scenarios for cloud execution and local execution, and manage expected values (screenshots) appropriate for each environment. Additionally, during cloud execution, if the expected font is not included in the font families installed on the device used, the text may be displayed in an unintended font. You cannot specify a font or verify whether it is displayed in the expected font itself.