The Generate Test Cases feature of Nexus AI agent generates test cases based on specifications or a text describing the behavior of the application under test.

Even with a very simple prompt like "Generate test cases from the attached file" it can produce a reasonably comprehensive set of test cases.

You can also adjust the granularity, comprehensiveness, and testing perspectives of the generated test cases by writing a specific prompt.

This document introduces the basic usage of the "Test Case Generation" feature and how to tune the output results.

1.Basic usage

Use suitable specifications as input

The type of testing that Nexus, as a test execution engine, is best at is user-perspective testing from the screen, also known as end-to-end (E2E) testing. Therefore, the tests that Nexus AI excels at generating are also E2E tests.

The most valuable documents for creating E2E tests are those that describe user behavior, such as Basic Design Documents, Overview Design Documents, and Screen Design Documents. Tickets in a ticket management system describing user behavior when a feature is added, or memos summarizing new feature overviews, are also suitable inputs.

On the other hand, information focusing on the system's mechanism rather than user behavior, such as Detailed Design Documents or Database Specifications, is not appropriate for creating E2E tests. Consequently, uploading such information is likely to result in insufficient test case output.

Try basic generation

Start by uploading a specification document you have and try generating test cases.

1. Open the Scenario Window, click the Generate with AI button, select Generate Test Cases from the displayed dropdown, and launch the "AI Agent" Window.

2. In the "AI Agent" Window, upload the specification document, enter the prompt "Generate test cases from the attached file", and click the "Send button".

3. The AI Agent will automatically generate test cases, and a CSV file will be saved in your download folder.

Evaluate the generated results

Evaluate the generated test cases from the following perspectives:

• Overall Granularity

  • Are the tests too detailed (e.g., valid but excessively minute or numerous) or too broad (e.g., mostly happy path with few semi-normal or error cases)?

• Accuracy

  • Is there hallucination (e.g., non-existent features or unnecessary tests)?

  • Are the anticipated testing perspectives comprehensively covered?

• Sufficiency/Usefulness of Rationale

  • Is the rationale for why each test case was produced correct and valid?

After evaluating these basic generation results, you can either use them to create tests or modify the prompt and tune the output to better match your desired test cases.

2.Adjust granularity

When you find that the output from the basic generation differs from your expectation, the most common discrepancy is in granularity.

Granularity differences can include:

• Expecting high-level, E2E tests, but getting detailed tests more akin to Integration Tests.

• Wanting scenario tests that cover the entire user story in one sequence, but getting tests that check only one perspective per scenario (or vice versa).

• Happy path test cases being generated for a certain item, but semi-normal or error cases being insufficiently covered or missing (or vice versa).

You can adjust this using the following two methods:

Specify the number of test cases

By explicitly specifying the number of test cases to be generated in the prompt, the granularity of the test cases will be adjusted accordingly.

For example, consider a case where the basic generation typically produces about 50 test cases.

If you want to generate more detailed tests, you can prompt:

Generate test cases from the attached file. At least 150 test cases are required.

By specifying a significantly larger number, the AI will attempt to generate tests with finer granularity and more detailed perspectives. In this case, the overall granularity of the generated test case set becomes finer.

Conversely, if too many cases are generated you can adjust their granularity by setting a maximum amount of test cases in the prompt.

Detailed specification

If a certain item, say Item A, is particularly important and you need more comprehensive test cases for it, you can adjust the test case granularity by providing a detailed specification for that item.

A detailed prompt can be effective:

Generate test cases from the attached file. For Item A, please cover all conceivable error cases in addition to the normal cases.

However, the above prompt does not guarantee the generation of absolutely exhaustive test cases. If necessary, you can use the outputted test cases as a base and expand them using other techniques (e.g., creating a coverage matrix using pairwise testing).

3.Improving accuracy

Even after matching the granularity, the generated test cases may not be exactly what you want.

In particular, you may need to suppress issues unique to generative AI, such as hallucination (e.g., non-existent features appearing, impossible combinations occurring) or the omission of necessary test cases. The following methods may be helpful:

Provide supplementary information

First, if high-level information such as the functions or testing perspectives that should be covered is missing, supplementing it manually can improve the accuracy of the subsequently generated test cases.

The attached file is the specification for Application A. It includes the following features: 
- Feature X
- Feature Y
- Feature Z
Based on this, please generate test cases from this specification.

While the example uses features X, Y, and Z, supplementing information like testing perspectives or uploading a separate file containing such information along with the specification document is also effective.

Narrow down the target for test case generation

Supplementing information like features or testing perspectives, as above, helps the AI deepen its understanding of the specification and generate better test cases.

However, as the input size increases, the AI tends to become less focused, and the density of test cases decreases. To avoid this, you can narrow down the target for test case generation to concentrate the AI's attention, thereby potentially improving the accuracy of the generated test cases.

The attached file is the specification for Application A. Please generate test cases for Feature X included in this specification

With this prompt, if there are three features: X, Y, and Z, only test cases for Feature X will be generated.

To cover all three, you would need to generate test cases for Feature Y in a separate session, and Feature Z in yet another session, and then concatenate them.

While this takes more effort, creating the tests by restricting the scope and generating three times, then concatenating is often more likely to produce better test cases than attempting to create tests for all features with a single prompt.

Note

• Generation results vary each time.

  • Due to the nature of generative AI, the results differ even with the same input. Try multiple times and select the highest quality test cases.

• Input size has limits.

  • There is a limit to the amount of text (token count) the AI can process at once. If using large documents, try to narrow down the scope.

• Document quality affects the results.

  • The quality of the input document (e.g., whether it is structured and well-organized) influences the outcome.