The Natural Language Step is an experimental feature, available only to customers whose contract includes the 'Nexus AI Agent' option.

Overview

Natural language step is a powerful feature that utilizes AI. Unlike regular steps, in a Natural language step, the AI interprets a prompt each time the test is run and executes operations according to the screen's state at that moment.

Natural language step is effective in the following cases:

Operations that use information displayed on the page
- Logging in using the username and password displayed on the screen.
- Reading and entering a confirmation code that is randomly generated each time.
Validation of elements with dynamically changing content
- Comparing and validating numbers in a table whose values change with each access.
Operations based on more abstract instructions
- Use AI to handle a scenario described as “Attempt to log in with invalid credentials."

Scenarios can be created entirely with Natural language steps, or by entrusting only specific parts to a Natural language step, such as dynamic calendar operations, and filling the rest of the scenario with regular steps.

How to create a Natural language step

Display the scenario editing screen and click Insert Step where you want to add a step.
Select Natural language step from the displayed options.
A modal for prompt input will be displayed, so enter the operation you want the AI to perform in natural language.
Once you have finished inputting, click Insert and then click Save to save the scenario.

Due to the nature of the feature, there is a possibility that the AI may not perform as expected in a Natural language step. We recommend adding an assertion step after the Natural language step to confirm that the expected state has been reached.

Prompt description examples

Prompts can be written flexibly, from a single operation to a test case containing multiple steps.

Prompting with a specific instruction

Enter test@example.com for the username and password123 for the password to log in, and then confirm that the text "Welcome, Test User" is displayed on the dashboard screen.

Prompting with a complete test case written using Gherkin format

Given: I am on the login page
When: I enter the correct credentials written on the page
And: I click the login button
Then: The login is successful

Checking execution results

When you execute a Natural language step, the series of operations actually performed by the AI is displayed as child steps on the test result details screen, as shown in the screenshot below.

Since the AI may use trial and error to find the optimal operation, some child steps may be recorded as failed. This is simply showing where the AI has tried different methods and does not necessarily mean the entire test has failed. Also, because the AI interprets the prompt each time the test is run, the content of the child steps may change with each execution.

If the test results are not as expected, we recommend modifying the prompt.

How to check the AI's reasoning

By adding an instruction to the prompt, you can have the AI output its reasoning so that you can check how it interpreted the prompt and performed the operation

For example, you could add a sentence like "Output the reason for the operations performed to the console log" at the end of the prompt.

This will record the information output by the AI during execution in the browser log, which can be found in the Log tab of the test result details screen. Use this information for debugging when the test does not behave as expected.

Constraints of Natural language steps

AI-based fixes: since this step does not record locators, the AI-based fixes feature is not supported.
Performance impact: interpreting natural language involves processing by a Large Language Model (LLM) and execution may take longer than regular steps. In many cases, the total execution time of a Natural language step will be about 10-20 seconds longer than the sum of the execution times of the AI-generated child steps.
Playwright code export: this step does not generate code, so it is excluded from the code export to Playwright.
Variable usage: it is not possible to use workspace variables or outputs from other steps within the prompt.

Convert natural language steps to no-code steps

In the Test result detail screen, you can save steps executed by a Natural language step as fixed no-code steps.

Converting Natural language steps to no-code steps may improve test reproducibility.

Instructions

Select the relevant test result from the list to open the Test result detail screen.
Check the step list and click the Convert to no-code steps button on the row of the Natural language step you want to convert.
Upon successful conversion, the message "Natural language step converted to no-code steps successfully." will be displayed.

Specifications and Behavior

When converted, the Scenario updates as follows:

Original Natural language step: remains in the list but is disabled (grayed out).
Generated steps: added immediately after the original step as new no-code steps.
Screenshots: each added step includes screenshot information from the execution.

The Convert to no-code steps button will be grayed out and disabled in the following cases:

If the original step is disabled. If you try anyway, a "Disabled natural language steps cannot be converted to no-code steps." error will happen.
If there are no successful generated steps: even if the Natural language step itself passed, you cannot convert it if none of the generated steps within it succeeded and a "This step has no successful generated steps to convert." error will happen.

Notes

Irreversible: once converted, you cannot revert to the original state.
- Please use Restoring a scenario from history to revert it to a previous version.
Successful steps only: only generated steps that succeeded during execution are converted and saved. Failed steps are excluded.
Impact on Shared groups: If the Natural language step is inside a Shared group, the Shared group itself is updated. The converted steps will be reflected in all scenarios referring to that Shared group.