The Natural Language Step is an experimental feature, available only to customers whose contract includes the 'Nexus AI Agent' option.
Overview
Natural Language Step is a powerful feature that utilizes AI. Unlike regular steps, in a Natural Language Step, the AI interprets a prompt each time the test is run and executes operations according to the screen's state at that moment.
This Natural Language Step is effective in the following cases:
Operations that use information displayed on the page
Logging in using the username and password displayed on the screen.
Reading and entering a confirmation code that is randomly generated each time.
Validation of elements with dynamically changing content
Comparing and validating numbers in a table whose values change with each access.
Operations based on more abstract instructions
Use AI to handle a scenario described as “Attempt to log in with invalid credentials."
Scenarios can be created entirely with Natural Language Steps, or by entrusting only specific parts to a Natural Language Step, such as dynamic calendar operations, and filling the rest of the scenario with regular steps.
How to create a Natural Language step
Display the scenario editing screen and click "Insert Step" where you want to add a step.
Select "Natural Language Step" from the displayed options.
A modal for prompt input will be displayed, so enter the operation you want the AI to perform in natural language.
Once you have finished inputting, click "Insert" and then click "Save" to save the scenario.
Due to the nature of the feature, there is a possibility that the AI may not perform as expected in a Natural Language Step. We recommend adding an assertion step after the Natural Language Step to confirm that the expected state has been reached.
Prompt description examples
Prompts can be written flexibly, from a single operation to a test case containing multiple steps.
Prompting with a specific instruction
Enter test@example.com for the username and password123 for the password to log in, and then confirm that the text "Welcome, Test User" is displayed on the dashboard screen.Prompting with a complete test case written using Gherkin format
Given: I am on the login page When: I enter the correct credentials written on the page And: I click the login button Then: The login is successful
How to check the AI's reasoning
By adding an instruction to the prompt, you can have the AI output its reasoning so that you can check how it interpreted the prompt and performed the operation
For example, you could add a sentence like "Output the reason for the operations performed to the console log" at the end of the prompt.
This will record the information output by the AI during execution in the browser log, which can be found in the Log tab of the test result details screen. Use this information for debugging when the test does not behave as expected.
Checking execution results
When you execute a Natural Language Step, the series of operations actually performed by the AI is displayed as child steps on the test result details screen, as shown in the screenshot below.
Since the AI may use trial and error to find the optimal operation, some child steps may be recorded as failed. This is simply showing where the AI has tried different methods and does not necessarily mean the entire test has failed. Also, because the AI interprets the prompt each time the test is run, the content of the child steps may change with each execution.
If the test results are not as expected, we recommend modifying the prompt.
Constraints of Natural Language Steps
AI-based Fixes: Since this step does not record locators, the AI-based Fixes feature is not supported.
Performance Impact: Interpreting natural language involves processing by a Large Language Model (LLM) and execution may take longer than regular steps. In many cases, the total execution time of a Natural Language Step will be about 10-20 seconds longer than the sum of the execution times of the AI-generated child steps.
Playwright Code Export: This step does not generate code, so it is excluded from the code export to Playwright.
Variable Usage: It is not possible to use workspace variables or outputs from other steps within the prompt.