CDRouter CLI Reference Guide

This document describes CDRouter’s CLI, which was previously referred to as buddy. The buddy command has been renamed cdrouter-cli. The original buddy command will be officially deprecated in future releases. For a short period of time the buddy command will continue to work, although all users are encouraged to update any scripts to use the new cdrouter-cli command instead. Note that the arguments associated with the cdrouter-cli command are compatible with the original buddy command.

For most users and testing tasks CDRouter’s web interface and its associated web CLI should be used. The CLI described in this document is the original CLI provided with CDRouter before the web component was added. This CLI should only be used by advanced users on an as needed basis. Please contact support@qacafe.com for more information.

Once the device under test (DUT) has been configured and connected to CDRouter, tests may be run using the CDRouter CLI. With no arguments, the cdrouter-cli command will run all available tests in the default test path of /usr/cdrouter/tests using default configuration settings.

[root@fox ~]# cdrouter-cli

INFO(setup): 15:05:26.981| Starting cdrouter-cli Tue Mar 22 15:05:25 EDT 2016
INFO(setup): 15:05:26.981| Copyright (c) 2001-2016 by QA Cafe
INFO(setup): 15:05:26.981| Version 10.0 build 1 (21342 trunk), built 2016-03-20 17:36:46 by nightly@cdr-forge6.lan (x86_64)
INFO(setup): 15:05:26.981| Loaded OS distro \S Kernel \r on an \m
INFO(setup): 15:05:26.981| Loaded OS version Linux-3.10.0-327.10.1.el7.x86_64 x86_64
INFO(setup): 15:05:26.981| Loaded Tcl version 8.6.4
INFO(setup): 15:05:26.981| Loaded buddy version 10.0.1
INFO(setup): 15:05:26.981| (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) )
INFO(setup): 15:05:26.981| Trying to load modules from '.'
INFO(setup): 15:05:26.981| Trying to load modules from '/usr/cdrouter/tests'
INFO(setup): 15:05:26.981| Start command: /usr/cdrouter/bin/cdrouter-cli -testvar lanInterface=enp0s8 -testvar wanInterface=enp0s9
INFO(setup): 15:05:26.981| Test Suite cdrouter 10.0.1
INFO(setup): 15:05:26.981| The system ID is 2df9e2a1f8c359183cf0191a20f2cc5a
INFO(setup): 15:05:26.981| Using license installed at: /etc/cdrouter.lic
INFO(setup): 15:05:26.981| Registered to: qacafe
INFO(setup): 15:05:26.981| Maintenance, Support and Upgrades until: 2016-05-11
INFO(setup): 15:05:26.981| Licensed to run: cdrouter
INFO(setup): 15:05:26.982| Multiport is yes
INFO(setup): 15:05:26.982| IPv6 is yes
INFO(setup): 15:05:26.982| Storage is yes
INFO(setup): 15:05:26.982| IKE is yes
INFO(setup): 15:05:26.982| TR69 is yes
INFO(setup): 15:05:26.982| TR69-EDM is yes
INFO(setup): 15:05:26.982| Nmap is yes
INFO(setup): 15:05:26.982| BBF.069 is yes
INFO(setup): 15:05:26.982| SNMP is yes
INFO(setup): 15:05:26.982| Performance is disabled
INFO(setup): 15:05:26.982| CPU is Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz, bogomips 5599.98
INFO(setup): 15:05:26.982| Loaded TclXML version 3.1 (libxml2), TclDOM 3.0, xmldefs 3.1
INFO(setup): 15:05:26.982| No configuration file specified.  Using default testvar settings.
INFO(setup): 15:05:26.982| Trying to load modules from '/usr/cdrouter/vendor/IOL/BBF.069/Tests'
INFO(setup): 15:05:26.982| BBF.069 version 1.2.075-11 (19093)

INFO(setup): 15:05:26.988| Running test start up procedure for Cable/DSL Router Test Suite
IMPORTANT(setup): 15:05:26.993| Starting CDRouter test setup
INFO(setup): 15:05:26.994| Loading Multiport test modules
INFO(setup): 15:05:26.997| Loading TR-069 test modules
INFO(setup): 15:05:26.999| Loading IKE test modules
INFO(setup): 15:05:26.999| Loading IPv6 test modules
INFO(setup): 15:05:27.002| Loading Storage test modules
INFO(setup): 15:05:27.003| Loading Nmap test modules
INFO(setup): 15:05:27.004| Loading BBF.069 test modules

INFO(setup): 15:05:27.004| Checking test configuration ...
INFO(setup): 15:05:27.005| IPv4 WAN configuration is 'DHCP'
INFO(setup): 15:05:27.005| DHCP address pool size is 6
INFO(setup): 15:05:27.005| Setting expected IPv4 TCP MSS size to 1460
INFO(setup): 15:05:27.006| Read 0 IPSEC tunnel configuration(s)
IMPORTANT(setup): 15:05:27.006| Restarting DUT ...

Specifying a Configuration File or Test Path

The -new-config option can be used to create a new configuration file.

Example:

# -- create a new test directory
$ mkdir /usr/cdrouter-data/custom/qalab/configs
$ cd /usr/cdrouter-data/custom/qalab/configs

# -- create a new configuration file
$ cdrouter-cli -new-config > myrouter.conf

# -- edit the configuration file to match your test environment
$ emacs myrouter.conf

To run any custom test cases that have been created, you must use the -testpath option to specify your custom test directory location.

Example:

# -- run all custom tests in the /usr/cdrouter-data/custom/qalab/labtest directory
$ cdrouter-cli -testpath "/usr/cdrouter/tests /usr/cdrouter-data/custom/qalab/labtest"

You can also setup defaults for the -testpath and -config options using shell environment variables. If no -testpath option is set, the value of the CDROUTER_CLI_TESTPATH variable in your environment will be used. Likewise, if no -config option is set, the value of the CDROUTER_CLI_CONFIG variable in your environment will be used. Note that when specifying testpaths the default testpath of /usr/cdrouter/tests must be included and it must be specified first.

Example:

$ export CDROUTER_CLI_TESTPATH="/usr/cdrouter/tests /usr/cdrouter-data/custom/qalab/labtest"
$ export CDROUTER_CLI_CONFIG=/usr/cdrouter-data/custom/qalab/configs/dhcp-wan.conf

Log Files

All test output from the CDRouter CLI is sent to standard out (stdout). You may use existing tools such as tee or script to capture the output. The CDRouter CLI also has built in options to create log files directly.

The -logfile option can be used to create a single log file. It takes a file name as an argument.

Example:

$ cdrouter-cli -logfile /usr/cdrouter-data/custom/lab/testrun/log.5.txt

A log directory may also be created using the -logdir option. When this option is used, a new directory is created for the test run. Each test case will generate its own log file. The output from the startup phase is placed in the start.txt file while the final results are placed in the final.txt file.

You can change the name of the log file format using the -number-format option. By default, the test case log is named using the test case name. The -force option can be used to delete any existing log directory before starting.

Example:

$ cdrouter-cli -logdir results -force
$ cdrouter-cli -logdir results -force -number-format

If test cases are repeated multiple times, additional test runs will contain a test instance number appended to the file name. For example:

$ cdrouter-cli -execute cdrouter_nat_201 -trace -pt -repeat -rcount 3 -logdir results -force
$ cd results
$ ls -l
total 88
-rw-r--r--    1 root     root        22189 Sep 28 12:30 cdrouter_nat_201.txt
-rw-r--r--    1 root     root        22189 Sep 28 12:30 cdrouter_nat_201.txt.2
-rw-r--r--    1 root     root        22189 Sep 28 12:30 cdrouter_nat_201.txt.3
-rw-r--r--    1 root     root          516 Sep 28 12:30 final.txt
-rw-r--r--    1 root     root        10175 Sep 28 12:30 start.txt

Capture Files

The CDRouter CLI can automatically generate capture files that contain all network traffic that is sent and received by CDRouter. One capture file is created for each physical interface under CDRouter’s control. The capture files can then be viewed using other tools such as CloudShark.

The capture feature provides another way to look at test results and debug network behavior. Some engineers may feel more comfortable dealing with capture files directly, instead of using the CDRouter log file.

To enable the creation of capture files, use the -capture option.

Example:

$ cdrouter-cli -capture -execute cdrouter_nat_1

By default, CDRouter will create files named start-wan.cap and start-lan.cap in the current working directory. If the -logdir option is used, the capture files will be placed in the log directory and individual capture files will be created for each test case.

Common Test Operations

There are several different test control methods including single test execution, looping, running until failures occur, and time based testing. The following are several common test operations:

With no arguments, all tests will be run using the default configuration file …
```
$ cdrouter-cli
```
If you want to run all tests and simultaneously display and capture the output to a log file …
```
$ cdrouter-cli | tee test.log
```
If you want to run all tests and display a one line summary for each packet sent and received …
```
$ cdrouter-cli -pt | tee test.log
```

If you want to run individual tests …

$ cdrouter-cli -execute cdrouter_nat_1,cdrouter_nat_2,cdrouter_nat_100

If you want to run all the tests in a specific expansion …
```
$ cdrouter-cli -expansion IPv6
```
If you want to run all the tests in one or more modules …
```
$ cdrouter-cli -module basic.tcl,nat.tcl
```
If you want to run all the tests in one module except “testName” …
```
$ cdrouter-cli -module <module.tcl> -skip-test <testName or #>
```
If you do not know what the test and execution numbers are …
```
$ cdrouter-cli -doc -module <moduleName.tcl>
```
If you want a summary of all the tests …
```
$ cdrouter-cli -summary
```
If you want to run a test with all possible tracing enabled …
```
$ cdrouter-cli -trace -decode -dump -execute <testName or #>
```

Please see the Advanced Test Execution section for additional test execution features and examples.

Understanding Test Messages

Depending on the tracing level used during a test run, various messages will be reported. Most test messages have a simple format. The general format is as follows:

Message-Type ( Location ): Timestamp| Text of message

The Message-Type can be either INFO, WARNING, PASS or FAIL. Most messages reported are INFO messages to explain what is happening during the test. WARNING messages are reported by the protocol state machines when an error or unexpected event happens. The PASS and FAIL messages are used to indicate that some part of a test passed or failed.

The Location indicates where the message originated. Messages are associated with different parts of the test. During the initial setup of the test suite, messages will be associated with ‘setup’. For example,

INFO(setup): 14:01:07| starting DHCP client on LAN interface eth1

During an actual test, messages that originate from the test case are associated directly with the test case number.

INFO(cdrouter-3): 14:08:44| starting new DHCP client

When an event happens on a specific interface, the message is associated with the specific protocol stack on that interface. For example, if an ARP packet is received on the LAN interface, you might see the following message.

INFO(lan): 14:08:45| ARP request, who is 192.168.1.3, tell 192.168.1.3

Protocol Decodes

To see full protocol decodes of the packets sent and received, you can specify the -decode option. The fully decoded packet will be displayed along with the sending or receiving interface.

PACKET-IN(eth1): received eth proto type 0800 
ETH: ---- Ethernet Packet (length = 590) ----
ETH: 
ETH: Source                     = 00:20:78:d2:bf:a5
ETH: Destination                = ff:ff:ff:ff:ff:ff
ETH: Type                       = IPv4 (0x0800)
ETH:
IPv4: ---- IPv4 Header ----
IPv4: 
IPv4: Version                   = 4
IPv4: Header Length             = 5
IPv4: TOS byte                  = 0x00
IPv4: Total Length              = 576
IPv4: ID                        = 0x0001
IPv4: Flags/Offset              = 0x0000
IPv4:   Don't Fragment          = 0
IPv4:   More Fragments          = 0
IPv4:   Offset                  = 0
IPv4: TTL                       = 150
IPv4: Protocol                  = UDP (0x11)
IPv4: Checksum                  = 0x6103
IPv4: Src                       = 192.168.1.1
IPv4: Dest                      = 255.255.255.255
IPv4:
UDP: ---- UDP Packet ----
UDP: 
UDP: Source Port                = 67 
UDP: Destination Port           = 68 
UDP: Length                     = 556
UDP: Checksum                   = 0x525a
UDP:
DHCP: ---- DHCP Packet ----
DHCP: 
DHCP: Type                      = Reply (0x02)
DHCP: Hardware Type             = 0x01
DHCP: Hardware Address Len      = 0x06
DHCP: Hops                      = 0x00
DHCP: Transaction ID            = 0x00408289 (xid)
DHCP: Seconds                   = 0x0000
DHCP: Flags                     = 0x0000
DHCP: Client IP                 = 0.0.0.0 (ciaddr)
DHCP: Your IP                   = 192.168.1.2 (yiaddr)
DHCP: Server IP                 = 0.0.0.0 (siaddr)
DHCP: Relay IP                  = 0.0.0.0 (giaddr)
DHCP: Client HW Addr            = 00:d0:b7:6b:04:2c (chaddr)
DHCP: ---- Options ----
DHCP:
DHCP: Magic Cookie              = 99.130.83.99
DHCP: Message Type              = DHCPACK
DHCP: Mask                      = 255.255.255.0
DHCP: Routers
DHCP:   Router 1                = 192.168.1.1
DHCP: DNS Servers
DHCP:   DNS Server 1            = 1.1.1.1
DHCP: IP Address Lease Time     = 86400 seconds (24.0 hours)
DHCP: Server Identifier         = 192.168.1.1
DHCP:
DHCP: ---- End DHCP Packet ----

You may also specify the -pt or -ptc options to see a one line summary for each packet instead of a full packet decode. The output format of the one line summary is similar to CloudShark.

There are several techniques that can be used to increase your testing mileage with CDRouter.

Repeating Tests

You can setup a test run that repeats a single test or collection of tests. This is helpful to verify that the router continues to function correctly over time. The -repeat option forces CDRouter into repeat mode. There are several other options that control the type of repeating.

Examples:

Repeat all nat.tcl tests forever …

$ cdrouter-cli -module nat.tcl –repeat

Repeat a single test 100 times …

$ cdrouter-cli -execute cdrouter_scale_1 -repeat -rcount 100

Repeat all the nat tests until one fails …

$ cdrouter-cli -module nat.tcl -repeat -until-fail

Repeat the nat.tcl module 100 times or until it fails …

$ cdrouter-cli -module nat.tcl -repeat -rcount 100 -until-fail

Repeat all tests until 5 failures occur …
```
$ cdrouter-cli -repeat -max-fail 5
```

Running a Test Range

You can specify a range of test numbers using the -begin and -end options. This allows you to run a range of test numbers that may span multiple modules. You can also add -skip-test options to skip specific tests in the range.

Examples:

Run tests 100 to 110 …

$ cdrouter-cli -config my.conf -begin 100 -end 110

Run tests 100 to 110, but skip test 108 …

$ cdrouter-cli -begin 100 -end 110 -skip-test 108

Time Based Testing

You can configure the test run to execute for a specific duration of time. CDRouter will continue to run tests until the time duration has elapsed. When the time duration is over, CDRouter will complete the current test and then exit.

Example:

Run all NAT tests for 1 hour (3600 seconds) …

$ cdrouter-cli -module nat.tcl -repeat -duration 3600

By default, CDRouter will wait for the current test to finish before exiting when the time duration has elapsed. You can use the -duration-interrupt option to instead have CDRouter interrupt the current test and exit immediately when the time duration has elapsed:

Example:

Run all NAT tests for 1 hour (3600 seconds), interrupting current test …

$ cdrouter-cli -module nat.tcl -repeat -duration 3600 -duration-interrupt

Finally, by default CDRouter considers the -duration time duration elapsing an error. You can use the -duration-no-error option to tell CDRouter to not consider it an error.

Retry Testing

You can configure the test run to repeat a test case if there is a failure. The test will be retried immediately and only one final result is produced. An optional delay may be specified before the retry is attempted.

Examples:

Repeat each test 2 additional times if there is a failure …
```
  $ cdrouter-cli -module nat.tcl –retry 2
```
Repeat each test 1 time after a failure, but wait 60 seconds before trying the test again …
```
$ cdrouter-cli –module nat.tcl –retry 2 –rdelay 60
```

Skipping Tests or Modules

During your testing, you may run into problems where the router gets into a state where it cannot recover. If this becomes a road block to additional testing, you can skip these tests or modules in order to setup long duration test runs. The -skip-test and -skip-module options provide this type of control.

Examples:

Skip the dhcp server tests …
```
$ cdrouter-cli -skip-module dhcp-s.tcl
```

Skip a specific test case by name …

$ cdrouter-cli -skip-test cdrouter_basic_1

You may also add skip entries into your configuration file instead of listing them as command line options. The entries may be added on any new line in the configuration file using the following entries:

buddy::skip_module <module-name>
buddy::skip_test <testcase-name>

As an example, the following two lines could be added to the configuration file to produce the same results as the original example above.

buddy::skip_module dhcp-s.tcl
buddy::skip_test cdrouter_basic_1

NOTE: The buddy namespace in Tcl is used internally by CDRouter. Some commands may be exposed to users, such as the buddy::skip_module and buddy::skip_test commands above.

Tracing Options

There are a number of tracing options that can used to adjust the output display during test execution:

cdrouter-cli -trace - Display all protocol tracing log messages
cdrouter-cli -pt -Display a one line summary for all packets in/out
cdrouter-cli -ptc - Display a one line summary with color highlights
cdrouter-cli -decode - Display full decodes of all packets in/out
cdrouter-cli -dump - Display hex dumps of all packets in/out
cdrouter-cli -silent - Display limited output including PASS/FAIL status

Trace Filters

All protocol trace messages may be filtered by protocol. The following options are available:

cdrouter-cli -protocols - List all protocols
cdrouter-cli -show-proto - Include protocol in all trace messages
cdrouter-cli -hide - Exclude protocols x,y,z from trace message output
cdrouter-cli -show - Only show protocols x,y,z in trace message output

Displaying Test Summaries

The test summary and description for each test case in a suite can be displayed using the -doc option. If no other options are used, the -doc option will generate test case descriptions for all test cases. The -doc option can also be combined with the -module and -execute flags to generate test case descriptions for specific modules and or test cases.

Examples:

Display test case descriptions for all tests …
```
$ cdrouter-cli -doc
```
Display test case descriptions for the nat module …
```
$ cdrouter-cli -doc -module nat.tcl
```
Display one line summary for each test …
```
$ cdrouter-cli -summary
```

Similar to the -doc option, the -web option produces test case descriptions in HTML format for viewing with a web browser.

The -list option is also available to list each test module and the number of test cases in each module.

Interactive Testing

CDRouter supports interactive testing and test execution using the -pause option. When the -pause option is specified on the command line, CDRouter will prompt the user to enter a key before executing each test. This allows you to run each test one at a time and interact with your device between tests. As an example, the -pause option allows you to bring up your router, enable any debug features on the router, and then start the test.

When CDRouter is waiting for the user to continue, it maintains active connections for any WAN or LAN protocols that are used. For example, CDRouter will continue to respond to ICMP pings, PPP LCP Echos, DHCP Requests, etc.

Example:

>>> PRESS <enter> to start the next test (cdrouter_nat_1):
INFO(wan): 10:03:01| Received LCP Echo-Request (id 61)
INFO(wan): 10:03:01| Sending LCP Echo-Reply (id 61)
>>> PRESS <enter> to start the next test (cdrouter_nat_1):

The prompt message is repeated after five seconds if any protocol events have occurred. This is a reminder that a key must be entered to continue.

NOTE: If you enter the “q” key followed by the “enter” key at the pause prompt, test execution will stop immediately. If you enter the “e” key followed by “enter” key at the pause prompt, pause mode will be disabled and all future tests will run without the pause mode.

Slowing Down Test Execution

Normally, CDRouter runs each test back to back. However, it is possible to slow down the test execution by introducing a delay before each test. When the -delay option is used, a delay will be added before each test is executed.

Example:

$ cdrouter-cli -delay 10 -module scaling.tcl
  INFO: 08:05:24| Waiting 10 seconds before starting test ...

During the delay period, CDRouter will continue to process any protocol events that occur on the WAN or LAN interfaces. For example, it will maintain the PPPoE connection or respond to any DHCP lease requests.

Setting Testvars on the Command-Line

Normally, the value of a testvar is set in the configuration file. However, you can also override the configuration file setting using the -testvar <name>=<value> command-line option.

Examples:

$ cdrouter-cli -config mysetup.conf -testvar wanMode=PPPoE
$ cdrouter-cli -config mysetup.conf -testvar wanMode=DHCP
$ cdrouter-cli -config mysetup.conf -testvar foo="one two three"

You can also delete a testvar using the -delete-testvar <name> command-line option, which removes any explicitly configured value for the given testvar.

Examples:

$ cdrouter-cli -config mysetup.conf -delete-testvar wanMode
$ cdrouter-cli -config mysetup.conf -delete-testvar foo

With CDRouter-Multiport, you can also set the value of a testvar that belongs to a testvar group such as an additional WAN or LAN interface using the -testvar_group <group>:<name>=<value> command-line option:

$ cdrouter-cli -config mysetup.conf -testvar_group wan2:wanMode=PPPoE
$ cdrouter-cli -config mysetup.conf -testvar_group wan3:wanMode=DHCP
$ cdrouter-cli -config mysetup.conf -testvar_group lan2:foo="one two three"

To delete a testvar in a specific testvar group, you can use the -delete-testvar_group <group>:<name> command-line option:

$ cdrouter-cli -config mysetup.conf -delete-testvar_group wan2:wanMode
$ cdrouter-cli -config mysetup.conf -delete-testvar_group lan2:foo

Finally, you can create or delete a testvar group using the -create-group <group> and -delete-group <group> command-line options:

$ cdrouter-cli -config mysetup.conf -create-group wan2
$ cdrouter-cli -config mysetup.conf -delete-group lan2

Stopping a Test Run

When CDRouter is interrupted using a Control-C key combination, the current test is stopped and the final test summary is displayed. The current test is not counted in the final summary results.

The test process may also be stopped by sending a SIGINT signal to the tcl process running cdrouter-cli.

Example:

$ ps -auwx | grep cdrouter-cli
(find process id for tclsh /usr/bin/cdrouter-cli)
$ kill -INT 3232

NOTE: If the cdrouter-cli process is being piped to another Linux command, the Control-C from the keyboard may not work. Some Linux tools have an option for ignoring signals and allowing the cdrouter-cli process to catch the signal directly. For example, the tee command can be run with the -i option to allow Control-C to work.

The PASS or FAIL status for each test case can be viewed and accessed in many different ways. At the end of a test run, the test status for each test case is displayed in the normal test output as a log message. There are additional output formats and programmatic interfaces available for accessing test results.

HTML Test Report

A HTML test report can be generated using the cdrouter-cli -report-html option from the command line. This option takes a filename argument which must be the name of a writable HTML file that will be created by CDRouter.

Example:

$ cdrouter-cli -report-html /usr/cdrouter-data/custom/user/reports/testrun_july_21.htm

The HTML output does not include any HTML header tags, images, or link references. The resulting HTML document may be easily included within other HTML documents.

The HTML report is created after all of the tests have completed. It is not possible to look at a partial HTML report while the tests are executing.

NOTE: If the cdrouter-cli -logdir option is also used with cdrouter-cli -report-html, the HTML report will contain links to each individual logfile.

Example:

$ cdrouter-cli -report-html ~root/testrun_july_21.htm -logdir ~root/logs –force

Exit Status

The exit status of the CDRouter CLI process can be used as a simple check to determine if any test cases failed during the test run. When test case failures occur, cdrouter-cli will exit with a non-zero exit status. If no failures occur, the exit status will be zero (0). When failures do occur, the exit status will reflect the actual number of test case failures up to 255 failures. If more than 255 test case failures occur, the exit status remains at 255.

The following bash script is example of using the exit status how to determine if one or more test cases failed.

Example:

#!/bin/bash
#
# Example script to check cdrouter-cli exit status from /bin/bash
#

# -- run the test suite
cdrouter-cli -config r.conf -module nat.tcl \> testresults.txt

# -- capture the exit status
STATUS=$?

# -- check the exit status
if [ $STATUS -gt 0 ]; then
  echo "At least one test case failed"
else
  echo "All test cases passed"
fi

Report Templates

Custom reports can be generated using the report template feature of CDRouter. This feature enables CDRouter to write a report record for each test case that is executed.

The record format for each result may be customized in any number of ways using several keywords. By combining the keywords with other formatting, template files can be created to generate CSV (Comma Separated Values), HTML, or XML data files containing test results. These results can then be imported into other applications or processed by other tools. The report file is updated after each test is executed so it is possible to have other tools dynamically reading the report file as tests execute.

To enable CDRouter to use report templates, the name of the report output file must be specified using the cdrouter-cli -report-template option. An additional file path must be specified for the template using the cdrouter-cli -template option.

Creating Template Files

A template file is used to setup the formatting for each result record that will be written to the report file. Several keywords are available that are dynamically expanded at run time to capture the test results. The following keywords are available:

Keyword	Description
`%TESTSUITE_NAME%`	This value is expanded to the test suite name such as CDRouter.
`%TESTSUITE_VERSION%`	This value is expanded to the test suite version number.
`%TESTCASE_NAME%`	This value is expanded to the test case name such as cdrouter_nat_1.
`%TESTCASE_NUMBER%`	This value is expanded to the test case execution number.
`%TESTCASE_MODULE%`	This value is expanded to the test module name that contains the test case such as nat.tcl, firewall.tcl, etc.
`%TESTCASE_RESULT%`	This value is expanded to the final result of the test case. The value is either PASS or FAIL.
`%TESTCASE_FAILREASON%`	This value is expanded to the first “FAIL” message string in the test case log. If the `buddy -retry` option is used, this is the message from the most recent failure if none of the attempts result in a PASS.
`%TESTCASE_FAILREASON_FIRSTTRY%`	If the `buddy -retry` option is used, this value is expanded to the “FAIL” message string of the first attempt in the test case log, regardless of whether the overall test result is PASS or FAIL.
`%TESTCASE_STARTTIME%`	This value is expanded to the time value in seconds when the test was started.
`%TESTCASE_STARTTIME_MSEC%`	This value is expanded to the time value in milliseconds when the test was started.
`%TESTCASE_STARTTIME_STR%`	This value is expanded to the test starting time as a displayable text string.
`%TESTCASE_STOPTIME%`	This value is expanded to the time value in seconds when the test finished.
`%TESTCASE_STOPTIME_MSEC%`	This value is expanded to the time value in milliseconds when the test finished.
`%TESTCASE_STOPTIME_STR%`	This value is expanded to the test stopping time as a displayable text string.
`%TESTCASE_DURATION%`	This value is expanded to the total duration of the test case in seconds. It is computed by subtracting the TESTCASE_STARTTIME from TESTCASE_STOPTIME.
`%TESTCASE_DESCRIPTION%`	This value is expanded to the one line test case description/synopsis.
`%TESTCASE_ATTEMPTS%`	This value contains the total number of times a test case is attempted to produce the current result. The total number of attempts is always 1 unless the `buddy -retry` option is being used.
`%TESTCASE_LOGFILE%`	This value is expanded to the name of the log file for the test case when either the `buddy -logfile` or `buddy -logdir` options are used.
`%TESTCASE_LOGDIR%`	This value is expanded to the name of the log directory when the `buddy -logdir` option is used.

Creating CSV Templates

A template file for CSV files can be created by placing the desired keywords in the template file separated by the “,” character (or other delimiter).

Example:

$ cat mycsv
%TESTCASE_NAME%,%TESTCASE_RESULT%
$ cdrouter-cli -report-template report.txt -template mycsv

In the example above, CDRouter will use the template file mycsv to generate the file “report.txt”. The data written to “report.txt” will be in CSV format.

Creating XML Templates

A template file for XML files can be created by placing keywords in a the template file along with XML tags. The XML template will be used to create a report containing XML data for each result.

Example:

$ cat myxml
<testcaseResult>
  <name>%TESTCASE_NAME%</name>
  <number>%TESTCASE_NUMBER%</number>
  <status>%TESTCASE_RESULT%</status>
  <duration>%TESTCASE_DURATION%</duration>
</testcaseResult>
$ cdrouter-cli -report-template report.txt -template myxml

Following an upgrade you may notice that the numbering of tests in the new version has changed. Depending on where the new tests were added, the execution number of the test may have changed. However, the test name never changes. The name of the test is displayed when the test runs, but you can also find the name using the cdrouter-cli -summary command line option. You can also use the test name any place where you use a test number.

Example:

Run a test using a test name

$ cdrouter-cli -execute cdrouter_basic_1

Run the same test using its execution number
```
$ cdrouter-cli -execute 1
```

Skip a test using its name

$ cdrouter-cli -skip-test cdrouter_basic_1

Skip the same test using its execution number
```
$ cdrouter-cli -skip-test 1
```

Information for all of the cdrouter-cli options can be displayed at any point using the help option:

$ cdrouter-cli -help
/usr/cdrouter/bin/cdrouter-cli [option [value]] ...
  -info                         Display test suite and host information
  -silent                       Display limited login and verification checkpoints
  -trace                        Display all protocol stack log output
  -include-proto                Include the protocol module in all trace messages
  -hide x,y,z                   Filter out trace messages for specific protocols
  -show x,y,z                   Only show trace messages for specific protocols
  -protocols                    List all registered protocols available for filters
  -pt                           Show one line summary for all packets sent and received
  -decode                       Display decodes of all packets
  -dump                         Display hex dumps of all packets
  -config x                     Set the config file name
  -new-config                   Print new config file and exit
  -upgrade-config               Print upgraded -config file and exit
  -logfile x                    Specify an output logfile
  -print-skipped                Print list of skipped tests based on config file and exit
  -check-config                 Check the config file for errors
  -skip-log                     Create logfile skip.log containing reasons for skipped tests
  -logdir x                     Specify an output log directory
  -force                        Automatically remove any existing log directory when starting
  -number-format                Create test log names based on the test execution number
  -capture                      Automatically generate capture files
  -testvar x=y                  Specify a testvar variable on the commandline to set
  -delete-testvar x             Specify a testvar variable on the commandline to delete
  -testvar_group group:x=y      Specify a testvar group variable on the commandline to set
  -delete-testvar_group group:x Specify a testvar group variable on the commandline to delete
  -create-group group           Specify a testvar group on the commandline to create (i.e. remove IGNORE)
  -delete-group group           Specify a testvar group on the commandline to delete (i.e. add IGNORE)
  -show-vars                    Show all test variables as they are set
  -addon x                      Same as -expansion, kept for backwards compatibility
  -expansion x                  Specify a specific expansion to execute
  -module x                     Specify a specific module to execute
  -execute x                    Execute a specific test name/number
  -skip-test x                  Skip execution of a specific test number
  -skip-module x                Skip execution of a test module
  -repeat                       Repeat all tests forever (see -until-fail)
  -rcount x                     Repeat all tests this many times (in a loop)
  -scount x                     Repeat each test this many times before proceeding
  -retry x                      Retry up to x times if test fails
  -rdelay x                     Add x second delay before retrying a failed test
  -shuffle                      Run the list of test cases in a random order
  -seed x                       Set the seed used with the -shuffle option
  -until-fail                   Stop once a test failure has occurred
  -max-fail x                   Stop when the failure count reaches x
  -begin x                      Begin executing tests from a specific test name/number
  -end x                        Stop executing tests at a specific test name/number
  -stop-after x                 Stop executing after running x tests (x >=0)
  -duration sec                 Run tests for x seconds and then stop
  -duration-interrupt           Interrupt currently executing test when -duration timeout elapses
  -duration-no-error            Do not consider -duration timeout elapsing an error
  -pause                        Prompt the user before starting each test
  -delay x                      Add x second delay before each test
  -list                         List test counts for each module
  -doc                          Print out documentation for the test case(s)
  -web                          Print out documentation in web format (HTML)
  -csv                          Print out documentation in CSV format
  -xml                          Print out documentation in XML format
  -summary                      Print out test case summaries only
  -xterm                        Display the test and module as the xterm title
  -xterm-stats                  Display the running count of tests as the xterm title
  -color                        Display pass/fail results in color
  -testpath                     Set the path to search for module files
  -no-control-c                 Display the Control-C handler
  -show-system-id               Display the System ID
  -update-license               Updates the QA Cafe license from qacafe.com
  -check-version                Checks for new version from qacafe.com
  -help                         Display this help
  -version                      Display the script version

Reporting options:
  -report-html     <file>       Generate an HTML report in <file>
  -report-template <file>       Generate a report based on a template in <file>
  -template        <file>       Specify the name of a template file
  -comment         <note>       Add a comment line to the log and HTML report

Example:
  cdrouter-cli -module nat.tcl -trace -xterm-stats -rcount 10

Product

Search Results

CDRouter CLI Reference Guide

Specifying a Configuration File or Test Path

Log Files

Capture Files

Common Test Operations

Understanding Test Messages

Protocol Decodes

Repeating Tests

Running a Test Range

Time Based Testing

Retry Testing

Skipping Tests or Modules

Tracing Options

Trace Filters

Displaying Test Summaries

Interactive Testing

Slowing Down Test Execution

Setting Testvars on the Command-Line

Stopping a Test Run

HTML Test Report

Exit Status

Report Templates

Creating Template Files

Creating CSV Templates

Creating XML Templates