How can I automatically retry failed tests?

Overview

One of the most common reasons for a transient test failure is packet loss on the device under test (DUT). Since CDRouter places a relatively low traffic load on a device, packet loss during a test session is typically rare. However, many CDRouter test cases are not tolerant of packet loss, and dropping a single packet may produce a test failure.

In test situations where packet loss may occur, the retry feature can be enabled to repeat a test automatically. When enabling the retry feature, the number of additional retries must also be configured. In most cases, the retry count should be a low number such as 1, 2, or 3. High retry counts could hide more serious problems where a test is failing consistently.

The retry feature also allows the user to configure a delay before the test will be retried. This could be used to allow the device under test to settle back to a normal operating state before attempting the test again.

Retry Feature Flowchart

Here’s a basic flowchart describing the test retry mechanism. The retry and rdelay variables are user specified and define the number of test retry attempts the delay between attempts, respectively.

Enabling the Retry Feature

The retry feature is enabled by editing the test package in the CDRouter web interface and clicking the “Edit Options” button.

From the cdrouter-cli command-line, the retry feature is enabled by passing in the -retry option along with the optional -rdelay x option.

cdrouter-cli -config mycpe.conf -module nat.tcl -retry 3 -rdelay 60

Viewing Test Results

When a test is retried, CDRouter treats the original test and all retries as a single test run. This means there will be only one final test result, one log file and one set of capture files. The log file will indicate if a retry has been attempted.

Detecting Retries with Template Results

The total number of test attempts can exported using the %TESTCASE_ATTEMPTS% keyword in any report template files. Please see the CDRouter User Guide for additional information on exporting test results with template files.

Applications

The new retry feature is perfect for developing a set of test packages or scenarios in which the expected result is all pass. By periodically executing the all pass test packages, issues can be quickly and easily identified if there are ANY failures in the final results. The retry feature can enhance this process by minimizing intermittent failures which may occur due to random packet loss. When the retry feature is enabled, any failures in the final results indicate that a particular test case or piece of functionality has consistently failed and may warrant further investigation.