CDRouter CLI Reference Guide
This document describes CDRouter’s CLI, which was previously referred to as
buddy
. The buddy
command has been renamed cdrouter-cli
. The original
buddy
command will be officially deprecated in future releases. For a short
period of time the buddy
command will continue to work, although all users are
encouraged to update any scripts to use the new cdrouter-cli
command instead.
Note that the arguments associated with the cdrouter-cli
command are
compatible with the original buddy
command.
For most users and testing tasks CDRouter’s web interface and its associated web CLI should be used. The CLI described in this document is the original CLI provided with CDRouter before the web component was added. This CLI should only be used by advanced users on an as needed basis. Please contact support@qacafe.com for more information.
Once the device under test (DUT) has been configured and connected to CDRouter,
tests may be run using the CDRouter CLI. With no arguments, the cdrouter-cli
command will run all available tests in the default test path of
/usr/cdrouter/tests
using default configuration settings.
[root@fox ~]# cdrouter-cli
INFO(setup): 15:05:26.981| Starting cdrouter-cli Tue Mar 22 15:05:25 EDT 2016
INFO(setup): 15:05:26.981| Copyright (c) 2001-2016 by QA Cafe
INFO(setup): 15:05:26.981| Version 10.0 build 1 (21342 trunk), built 2016-03-20 17:36:46 by nightly@cdr-forge6.lan (x86_64)
INFO(setup): 15:05:26.981| Loaded OS distro \S Kernel \r on an \m
INFO(setup): 15:05:26.981| Loaded OS version Linux-3.10.0-327.10.1.el7.x86_64 x86_64
INFO(setup): 15:05:26.981| Loaded Tcl version 8.6.4
INFO(setup): 15:05:26.981| Loaded buddy version 10.0.1
INFO(setup): 15:05:26.981| (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) )
INFO(setup): 15:05:26.981| Trying to load modules from '.'
INFO(setup): 15:05:26.981| Trying to load modules from '/usr/cdrouter/tests'
INFO(setup): 15:05:26.981| Start command: /usr/cdrouter/bin/cdrouter-cli -testvar lanInterface=enp0s8 -testvar wanInterface=enp0s9
INFO(setup): 15:05:26.981| Test Suite cdrouter 10.0.1
INFO(setup): 15:05:26.981| The system ID is 2df9e2a1f8c359183cf0191a20f2cc5a
INFO(setup): 15:05:26.981| Using license installed at: /etc/cdrouter.lic
INFO(setup): 15:05:26.981| Registered to: qacafe
INFO(setup): 15:05:26.981| Maintenance, Support and Upgrades until: 2016-05-11
INFO(setup): 15:05:26.981| Licensed to run: cdrouter
INFO(setup): 15:05:26.982| Multiport is yes
INFO(setup): 15:05:26.982| IPv6 is yes
INFO(setup): 15:05:26.982| Storage is yes
INFO(setup): 15:05:26.982| IKE is yes
INFO(setup): 15:05:26.982| TR69 is yes
INFO(setup): 15:05:26.982| TR69-EDM is yes
INFO(setup): 15:05:26.982| Nmap is yes
INFO(setup): 15:05:26.982| BBF.069 is yes
INFO(setup): 15:05:26.982| SNMP is yes
INFO(setup): 15:05:26.982| Performance is disabled
INFO(setup): 15:05:26.982| CPU is Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz, bogomips 5599.98
INFO(setup): 15:05:26.982| Loaded TclXML version 3.1 (libxml2), TclDOM 3.0, xmldefs 3.1
INFO(setup): 15:05:26.982| No configuration file specified. Using default testvar settings.
INFO(setup): 15:05:26.982| Trying to load modules from '/usr/cdrouter/vendor/IOL/BBF.069/Tests'
INFO(setup): 15:05:26.982| BBF.069 version 1.2.075-11 (19093)
INFO(setup): 15:05:26.988| Running test start up procedure for Cable/DSL Router Test Suite
IMPORTANT(setup): 15:05:26.993| Starting CDRouter test setup
INFO(setup): 15:05:26.994| Loading Multiport test modules
INFO(setup): 15:05:26.997| Loading TR-069 test modules
INFO(setup): 15:05:26.999| Loading IKE test modules
INFO(setup): 15:05:26.999| Loading IPv6 test modules
INFO(setup): 15:05:27.002| Loading Storage test modules
INFO(setup): 15:05:27.003| Loading Nmap test modules
INFO(setup): 15:05:27.004| Loading BBF.069 test modules
INFO(setup): 15:05:27.004| Checking test configuration ...
INFO(setup): 15:05:27.005| IPv4 WAN configuration is 'DHCP'
INFO(setup): 15:05:27.005| DHCP address pool size is 6
INFO(setup): 15:05:27.005| Setting expected IPv4 TCP MSS size to 1460
INFO(setup): 15:05:27.006| Read 0 IPSEC tunnel configuration(s)
IMPORTANT(setup): 15:05:27.006| Restarting DUT ...
Specifying a Configuration File or Test Path
The -new-config
option can be used to create a new configuration file.
Example:
# -- create a new test directory
$ mkdir /usr/cdrouter-data/custom/qalab/configs
$ cd /usr/cdrouter-data/custom/qalab/configs
# -- create a new configuration file
$ cdrouter-cli -new-config > myrouter.conf
# -- edit the configuration file to match your test environment
$ emacs myrouter.conf
To run any custom test cases that have been created, you must use the
-testpath
option to specify your custom test directory location.
Example:
# -- run all custom tests in the /usr/cdrouter-data/custom/qalab/labtest directory
$ cdrouter-cli -testpath "/usr/cdrouter/tests /usr/cdrouter-data/custom/qalab/labtest"
You can also setup defaults for the -testpath
and -config
options using
shell environment variables. If no -testpath
option is set, the value of the
CDROUTER_CLI_TESTPATH variable in your environment will be used. Likewise, if no
-config
option is set, the value of the CDROUTER_CLI_CONFIG variable in your
environment will be used. Note that when specifying testpaths the default
testpath of /usr/cdrouter/tests
must be included and it must be specified
first.
Example:
$ export CDROUTER_CLI_TESTPATH="/usr/cdrouter/tests /usr/cdrouter-data/custom/qalab/labtest"
$ export CDROUTER_CLI_CONFIG=/usr/cdrouter-data/custom/qalab/configs/dhcp-wan.conf
Log Files
All test output from the CDRouter CLI is sent to standard out (stdout). You may use existing tools such as tee or script to capture the output. The CDRouter CLI also has built in options to create log files directly.
The -logfile
option can be used to create a single log file. It takes a file
name as an argument.
Example:
$ cdrouter-cli -logfile /usr/cdrouter-data/custom/lab/testrun/log.5.txt
A log directory may also be created using the -logdir
option. When this
option is used, a new directory is created for the test run. Each test case
will generate its own log file. The output from the startup phase is placed in
the start.txt
file while the final results are placed in the final.txt
file.
You can change the name of the log file format using the -number-format
option. By default, the test case log is named using the test case
name. The -force
option can be used to delete any existing log
directory before starting.
Example:
$ cdrouter-cli -logdir results -force
$ cdrouter-cli -logdir results -force -number-format
If test cases are repeated multiple times, additional test runs will contain a test instance number appended to the file name. For example:
$ cdrouter-cli -execute cdrouter_nat_201 -trace -pt -repeat -rcount 3 -logdir results -force
$ cd results
$ ls -l
total 88
-rw-r--r-- 1 root root 22189 Sep 28 12:30 cdrouter_nat_201.txt
-rw-r--r-- 1 root root 22189 Sep 28 12:30 cdrouter_nat_201.txt.2
-rw-r--r-- 1 root root 22189 Sep 28 12:30 cdrouter_nat_201.txt.3
-rw-r--r-- 1 root root 516 Sep 28 12:30 final.txt
-rw-r--r-- 1 root root 10175 Sep 28 12:30 start.txt
Capture Files
The CDRouter CLI can automatically generate capture files that contain all network traffic that is sent and received by CDRouter. One capture file is created for each physical interface under CDRouter’s control. The capture files can then be viewed using other tools such as CloudShark.
The capture feature provides another way to look at test results and debug network behavior. Some engineers may feel more comfortable dealing with capture files directly, instead of using the CDRouter log file.
To enable the creation of capture files, use the -capture
option.
Example:
$ cdrouter-cli -capture -execute cdrouter_nat_1
By default, CDRouter will create files named start-wan.cap
and start-lan.cap
in the current working directory. If the -logdir
option is used, the
capture files will be placed in the log directory and individual
capture files will be created for each test case.
Common Test Operations
There are several different test control methods including single test execution, looping, running until failures occur, and time based testing. The following are several common test operations:
-
With no arguments, all tests will be run using the default configuration file …
$ cdrouter-cli
-
If you want to run all tests and simultaneously display and capture the output to a log file …
$ cdrouter-cli | tee test.log
-
If you want to run all tests and display a one line summary for each packet sent and received …
$ cdrouter-cli -pt | tee test.log
-
If you want to run individual tests …
$ cdrouter-cli -execute cdrouter_nat_1,cdrouter_nat_2,cdrouter_nat_100
-
If you want to run all the tests in a specific addon …
$ cdrouter-cli -addon IPv6
-
If you want to run all the tests in one or more modules …
$ cdrouter-cli -module basic.tcl,nat.tcl
-
If you want to run all the tests in one module except “testName” …
$ cdrouter-cli -module <module.tcl> -skip-test <testName or #>
-
If you do not know what the test and execution numbers are …
$ cdrouter-cli -doc -module <moduleName.tcl>
-
If you want a summary of all the tests …
$ cdrouter-cli -summary
-
If you want to run a test with all possible tracing enabled …
$ cdrouter-cli -trace -decode -dump -execute <testName or #>
Please see the Advanced Test Execution section for additional test execution features and examples.
Understanding Test Messages
Depending on the tracing level used during a test run, various messages will be reported. Most test messages have a simple format. The general format is as follows:
Message-Type ( Location ): Timestamp| Text of message
The Message-Type can be either INFO, WARNING, PASS or FAIL. Most messages reported are INFO messages to explain what is happening during the test. WARNING messages are reported by the protocol state machines when an error or unexpected event happens. The PASS and FAIL messages are used to indicate that some part of a test passed or failed.
The Location indicates where the message originated. Messages are associated with different parts of the test. During the initial setup of the test suite, messages will be associated with ‘setup’. For example,
INFO(setup): 14:01:07| starting DHCP client on LAN interface eth1
During an actual test, messages that originate from the test case are associated directly with the test case number.
INFO(cdrouter-3): 14:08:44| starting new DHCP client
When an event happens on a specific interface, the message is associated with the specific protocol stack on that interface. For example, if an ARP packet is received on the LAN interface, you might see the following message.
INFO(lan): 14:08:45| ARP request, who is 192.168.1.3, tell 192.168.1.3
Protocol Decodes
To see full protocol decodes of the packets sent and received, you can specify
the -decode
option. The fully decoded packet will be displayed along with the
sending or receiving interface.
PACKET-IN(eth1): received eth proto type 0800
ETH: ---- Ethernet Packet (length = 590) ----
ETH:
ETH: Source = 00:20:78:d2:bf:a5
ETH: Destination = ff:ff:ff:ff:ff:ff
ETH: Type = IPv4 (0x0800)
ETH:
IPv4: ---- IPv4 Header ----
IPv4:
IPv4: Version = 4
IPv4: Header Length = 5
IPv4: TOS byte = 0x00
IPv4: Total Length = 576
IPv4: ID = 0x0001
IPv4: Flags/Offset = 0x0000
IPv4: Don't Fragment = 0
IPv4: More Fragments = 0
IPv4: Offset = 0
IPv4: TTL = 150
IPv4: Protocol = UDP (0x11)
IPv4: Checksum = 0x6103
IPv4: Src = 192.168.1.1
IPv4: Dest = 255.255.255.255
IPv4:
UDP: ---- UDP Packet ----
UDP:
UDP: Source Port = 67
UDP: Destination Port = 68
UDP: Length = 556
UDP: Checksum = 0x525a
UDP:
DHCP: ---- DHCP Packet ----
DHCP:
DHCP: Type = Reply (0x02)
DHCP: Hardware Type = 0x01
DHCP: Hardware Address Len = 0x06
DHCP: Hops = 0x00
DHCP: Transaction ID = 0x00408289 (xid)
DHCP: Seconds = 0x0000
DHCP: Flags = 0x0000
DHCP: Client IP = 0.0.0.0 (ciaddr)
DHCP: Your IP = 192.168.1.2 (yiaddr)
DHCP: Server IP = 0.0.0.0 (siaddr)
DHCP: Relay IP = 0.0.0.0 (giaddr)
DHCP: Client HW Addr = 00:d0:b7:6b:04:2c (chaddr)
DHCP: ---- Options ----
DHCP:
DHCP: Magic Cookie = 99.130.83.99
DHCP: Message Type = DHCPACK
DHCP: Mask = 255.255.255.0
DHCP: Routers
DHCP: Router 1 = 192.168.1.1
DHCP: DNS Servers
DHCP: DNS Server 1 = 1.1.1.1
DHCP: IP Address Lease Time = 86400 seconds (24.0 hours)
DHCP: Server Identifier = 192.168.1.1
DHCP:
DHCP: ---- End DHCP Packet ----
You may also specify the -pt
or -ptc
options to see a one line summary for
each packet instead of a full packet decode. The output format of the one line
summary is similar to CloudShark.
There are several techniques that can be used to increase your testing mileage with CDRouter.
Repeating Tests
You can setup a test run that repeats a single test or collection of
tests. This is helpful to verify that the router continues to function
correctly over time. The -repeat
option forces CDRouter into repeat mode.
There are several other options that control the type of repeating.
Examples:
-
Repeat all nat.tcl tests forever …
$ cdrouter-cli -module nat.tcl –repeat
-
Repeat a single test 100 times …
$ cdrouter-cli -execute cdrouter_scale_1 -repeat -rcount 100
-
Repeat all the nat tests until one fails …
$ cdrouter-cli -module nat.tcl -repeat -until-fail
-
Repeat the nat.tcl module 100 times or until it fails …
$ cdrouter-cli -module nat.tcl -repeat -rcount 100 -until-fail
-
Repeat all tests until 5 failures occur …
$ cdrouter-cli -repeat -max-fail 5
Running a Test Range
You can specify a range of test numbers using the -begin
and -end
options.
This allows you to run a range of test numbers that may span multiple modules.
You can also add -skip-test
options to skip specific tests in the range.
Examples:
-
Run tests 100 to 110 …
$ cdrouter-cli -config my.conf -begin 100 -end 110
-
Run tests 100 to 110, but skip test 108 …
$ cdrouter-cli -begin 100 -end 110 -skip-test 108
Time Based Testing
You can configure the test run to execute for a specific duration of time. CDRouter will continue to run tests until the time duration has elapsed. When the time duration is over, CDRouter will complete the current test and then exit.
Example:
-
Run all NAT tests for 1 hour (3600 seconds) …
$ cdrouter-cli -module nat.tcl -repeat -duration 3600
By default, CDRouter will wait for the current test to
finish before exiting when the time duration has elapsed. You can use
the -duration-interrupt
option to instead have CDRouter interrupt
the current test and exit immediately when the time duration has
elapsed:
Example:
-
Run all NAT tests for 1 hour (3600 seconds), interrupting current test …
$ cdrouter-cli -module nat.tcl -repeat -duration 3600 -duration-interrupt
Finally, by default CDRouter considers the -duration
time duration
elapsing an error. You can use the -duration-no-error
option to
tell CDRouter to not consider it an error.
Retry Testing
You can configure the test run to repeat a test case if there is a failure. The test will be retried immediately and only one final result is produced. An optional delay may be specified before the retry is attempted.
Examples:
-
Repeat each test 2 additional times if there is a failure …
$ cdrouter-cli -module nat.tcl –retry 2
-
Repeat each test 1 time after a failure, but wait 60 seconds before trying the test again …
$ cdrouter-cli –module nat.tcl –retry 2 –rdelay 60
Skipping Tests or Modules
During your testing, you may run into problems where the router gets
into a state where it cannot recover. If this becomes a road block to
additional testing, you can skip these tests or modules in order to
setup long duration test runs. The -skip-test
and -skip-module
options
provide this type of control.
Examples:
-
Skip the dhcp server tests …
$ cdrouter-cli -skip-module dhcp-s.tcl
-
Skip a specific test case by name …
$ cdrouter-cli -skip-test cdrouter_basic_1
You may also add skip entries into your configuration file instead of listing them as command line options. The entries may be added on any new line in the configuration file using the following entries:
buddy::skip_module <module-name>
buddy::skip_test <testcase-name>
As an example, the following two lines could be added to the configuration file to produce the same results as the original example above.
buddy::skip_module dhcp-s.tcl
buddy::skip_test cdrouter_basic_1
NOTE: The buddy
namespace in Tcl is used internally by CDRouter. Some
commands may be exposed to users, such as the buddy::skip_module
and
buddy::skip_test
commands above.
Tracing Options
There are a number of tracing options that can used to adjust the output display during test execution:
cdrouter-cli -trace
- Display all protocol tracing log messagescdrouter-cli -pt
-Display a one line summary for all packets in/outcdrouter-cli -ptc
- Display a one line summary with color highlightscdrouter-cli -decode
- Display full decodes of all packets in/outcdrouter-cli -dump
- Display hex dumps of all packets in/outcdrouter-cli -silent
- Display limited output including PASS/FAIL status
Trace Filters
All protocol trace messages may be filtered by protocol. The following options are available:
cdrouter-cli -protocols
- List all protocolscdrouter-cli -show-proto
- Include protocol in all trace messagescdrouter-cli -hide
- Exclude protocols x,y,z from trace message outputcdrouter-cli -show
- Only show protocols x,y,z in trace message output
Displaying Test Summaries
The test summary and description for each test case in a suite can be
displayed using the -doc
option. If no other options are used,
the -doc
option will generate test case descriptions for all test cases.
The -doc
option can also be combined with the -module
and -execute
flags
to generate test case descriptions for specific modules and or test cases.
Examples:
-
Display test case descriptions for all tests …
$ cdrouter-cli -doc
-
Display test case descriptions for the nat module …
$ cdrouter-cli -doc -module nat.tcl
-
Display one line summary for each test …
$ cdrouter-cli -summary
Similar to the -doc
option, the -web
option produces test case descriptions
in HTML format for viewing with a web browser.
The -list
option is also available to list each test module and the number of
test cases in each module.
Interactive Testing
CDRouter supports interactive testing and test execution using the -pause
option. When the -pause
option is specified on the command line, CDRouter will
prompt the user to enter a key before executing each test. This allows you to
run each test one at a time and interact with your device between tests. As an
example, the -pause
option allows you to bring up your router, enable any
debug features on the router, and then start the test.
When CDRouter is waiting for the user to continue, it maintains active connections for any WAN or LAN protocols that are used. For example, CDRouter will continue to respond to ICMP pings, PPP LCP Echos, DHCP Requests, etc.
Example:
>>> PRESS <enter> to start the next test (cdrouter_nat_1):
INFO(wan): 10:03:01| Received LCP Echo-Request (id 61)
INFO(wan): 10:03:01| Sending LCP Echo-Reply (id 61)
>>> PRESS <enter> to start the next test (cdrouter_nat_1):
The prompt message is repeated after five seconds if any protocol events have occurred. This is a reminder that a key must be entered to continue.
NOTE: If you enter the “q” key followed by the “enter” key at the pause prompt, test execution will stop immediately. If you enter the “e” key followed by “enter” key at the pause prompt, pause mode will be disabled and all future tests will run without the pause mode.
Slowing Down Test Execution
Normally, CDRouter runs each test back to back. However, it is
possible to slow down the test execution by introducing a delay before
each test. When the -delay
option is used, a delay will be
added before each test is executed.
Example:
$ cdrouter-cli -delay 10 -module scaling.tcl
INFO: 08:05:24| Waiting 10 seconds before starting test ...
During the delay period, CDRouter will continue to process any protocol events that occur on the WAN or LAN interfaces. For example, it will maintain the PPPoE connection or respond to any DHCP lease requests.
Setting Testvars on the Command-Line
Normally, the value of a testvar is set in the configuration file.
However, you can also override the configuration file setting using
the -testvar <name>=<value>
command-line option.
Examples:
$ cdrouter-cli -config mysetup.conf -testvar wanMode=PPPoE
$ cdrouter-cli -config mysetup.conf -testvar wanMode=DHCP
$ cdrouter-cli -config mysetup.conf -testvar foo="one two three"
You can also delete a testvar using the -delete-testvar <name>
command-line option, which removes any explicitly configured value for
the given testvar.
Examples:
$ cdrouter-cli -config mysetup.conf -delete-testvar wanMode
$ cdrouter-cli -config mysetup.conf -delete-testvar foo
With CDRouter-Multiport, you can also set the value of a testvar that
belongs to a testvar group such as an additional WAN or LAN interface
using the -testvar_group <group>:<name>=<value>
command-line option:
$ cdrouter-cli -config mysetup.conf -testvar_group wan2:wanMode=PPPoE
$ cdrouter-cli -config mysetup.conf -testvar_group wan3:wanMode=DHCP
$ cdrouter-cli -config mysetup.conf -testvar_group lan2:foo="one two three"
To delete a testvar in a specific testvar group, you can use the
-delete-testvar_group <group>:<name>
command-line option:
$ cdrouter-cli -config mysetup.conf -delete-testvar_group wan2:wanMode
$ cdrouter-cli -config mysetup.conf -delete-testvar_group lan2:foo
Finally, you can create or delete a testvar group using the
-create-group <group>
and -delete-group <group>
command-line
options:
$ cdrouter-cli -config mysetup.conf -create-group wan2
$ cdrouter-cli -config mysetup.conf -delete-group lan2
Stopping a Test Run
When CDRouter is interrupted using a Control-C key combination, the current test is stopped and the final test summary is displayed. The current test is not counted in the final summary results.
The test process may also be stopped by sending a SIGINT signal to the tcl process running cdrouter-cli.
Example:
$ ps -auwx | grep cdrouter-cli
(find process id for tclsh /usr/bin/cdrouter-cli)
$ kill -INT 3232
NOTE: If the cdrouter-cli process is being piped to another Linux command, the Control-C from the keyboard may not work. Some Linux tools have an option for ignoring signals and allowing the cdrouter-cli process to catch the signal directly. For example, the tee command can be run with the -i option to allow Control-C to work.
The PASS or FAIL status for each test case can be viewed and accessed in many different ways. At the end of a test run, the test status for each test case is displayed in the normal test output as a log message. There are additional output formats and programmatic interfaces available for accessing test results.
HTML Test Report
A HTML test report can be generated using the cdrouter-cli -report-html
option
from the command line. This option takes a filename argument which must be the
name of a writable HTML file that will be created by CDRouter.
Example:
$ cdrouter-cli -report-html /usr/cdrouter-data/custom/user/reports/testrun_july_21.htm
The HTML output does not include any HTML header tags, images, or link references. The resulting HTML document may be easily included within other HTML documents.
The HTML report is created after all of the tests have completed. It is not possible to look at a partial HTML report while the tests are executing.
NOTE: If the cdrouter-cli -logdir
option is also used with cdrouter-cli -report-html
, the HTML report will contain links to each individual logfile.
Example:
$ cdrouter-cli -report-html ~root/testrun_july_21.htm -logdir ~root/logs –force
Exit Status
The exit status of the CDRouter CLI process can be used as a simple check to determine if any test cases failed during the test run. When test case failures occur, cdrouter-cli will exit with a non-zero exit status. If no failures occur, the exit status will be zero (0). When failures do occur, the exit status will reflect the actual number of test case failures up to 255 failures. If more than 255 test case failures occur, the exit status remains at 255.
The following bash script is example of using the exit status how to determine if one or more test cases failed.
Example:
#!/bin/bash
#
# Example script to check cdrouter-cli exit status from /bin/bash
#
# -- run the test suite
cdrouter-cli -config r.conf -module nat.tcl \> testresults.txt
# -- capture the exit status
STATUS=$?
# -- check the exit status
if [ $STATUS -gt 0 ]; then
echo "At least one test case failed"
else
echo "All test cases passed"
fi
Report Templates
Custom reports can be generated using the report template feature of CDRouter. This feature enables CDRouter to write a report record for each test case that is executed.
The record format for each result may be customized in any number of ways using several keywords. By combining the keywords with other formatting, template files can be created to generate CSV (Comma Separated Values), HTML, or XML data files containing test results. These results can then be imported into other applications or processed by other tools. The report file is updated after each test is executed so it is possible to have other tools dynamically reading the report file as tests execute.
To enable CDRouter to use report templates, the name of the report output file
must be specified using the cdrouter-cli -report-template
option. An
additional file path must be specified for the template using the cdrouter-cli -template
option.
Creating Template Files
A template file is used to setup the formatting for each result record that will be written to the report file. Several keywords are available that are dynamically expanded at run time to capture the test results. The following keywords are available:
Keyword | Description |
---|---|
%TESTSUITE_NAME% |
This value is expanded to the test suite name such as CDRouter. |
%TESTSUITE_VERSION% |
This value is expanded to the test suite version number. |
%TESTCASE_NAME% |
This value is expanded to the test case name such as cdrouter_nat_1. |
%TESTCASE_NUMBER% |
This value is expanded to the test case execution number. |
%TESTCASE_MODULE% |
This value is expanded to the test module name that contains the test case such as nat.tcl, firewall.tcl, etc. |
%TESTCASE_RESULT% |
This value is expanded to the final result of the test case. The value is either PASS or FAIL. |
%TESTCASE_FAILREASON% |
This value is expanded to the first “FAIL” message string in the test case log. If the buddy -retry option is used, this is the message from the most recent failure if none of the attempts result in a PASS. |
%TESTCASE_FAILREASON_FIRSTTRY% |
If the buddy -retry option is used, this value is expanded to the “FAIL” message string of the first attempt in the test case log, regardless of whether the overall test result is PASS or FAIL. |
%TESTCASE_STARTTIME% |
This value is expanded to the time value in seconds when the test was started. |
%TESTCASE_STARTTIME_MSEC% |
This value is expanded to the time value in milliseconds when the test was started. |
%TESTCASE_STARTTIME_STR% |
This value is expanded to the test starting time as a displayable text string. |
%TESTCASE_STOPTIME% |
This value is expanded to the time value in seconds when the test finished. |
%TESTCASE_STOPTIME_MSEC% |
This value is expanded to the time value in milliseconds when the test finished. |
%TESTCASE_STOPTIME_STR% |
This value is expanded to the test stopping time as a displayable text string. |
%TESTCASE_DURATION% |
This value is expanded to the total duration of the test case in seconds. It is computed by subtracting the TESTCASE_STARTTIME from TESTCASE_STOPTIME. |
%TESTCASE_DESCRIPTION% |
This value is expanded to the one line test case description/synopsis. |
%TESTCASE_ATTEMPTS% |
This value contains the total number of times a test case is attempted to produce the current result. The total number of attempts is always 1 unless the buddy -retry option is being used. |
%TESTCASE_LOGFILE% |
This value is expanded to the name of the log file for the test case when either the buddy -logfile or buddy -logdir options are used. |
%TESTCASE_LOGDIR% |
This value is expanded to the name of the log directory when the buddy -logdir option is used. |
Creating CSV Templates
A template file for CSV files can be created by placing the desired keywords in the template file separated by the “,” character (or other delimiter).
Example:
$ cat mycsv
%TESTCASE_NAME%,%TESTCASE_RESULT%
$ cdrouter-cli -report-template report.txt -template mycsv
In the example above, CDRouter will use the template file mycsv
to generate
the file “report.txt”. The data written to “report.txt” will be in CSV format.
Creating XML Templates
A template file for XML files can be created by placing keywords in a the template file along with XML tags. The XML template will be used to create a report containing XML data for each result.
Example:
$ cat myxml
<testcaseResult>
<name>%TESTCASE_NAME%</name>
<number>%TESTCASE_NUMBER%</number>
<status>%TESTCASE_RESULT%</status>
<duration>%TESTCASE_DURATION%</duration>
</testcaseResult>
$ cdrouter-cli -report-template report.txt -template myxml
Following an upgrade you may notice that the numbering of tests in the new
version has changed. Depending on where the new tests were added, the execution
number of the test may have changed. However, the test name never changes. The
name of the test is displayed when the test runs, but you can also find the name
using the cdrouter-cli -summary
command line option. You can also use the test
name any place where you use a test number.
Example:
-
Run a test using a test name
$ cdrouter-cli -execute cdrouter_basic_1
-
Run the same test using its execution number
$ cdrouter-cli -execute 1
-
Skip a test using its name
$ cdrouter-cli -skip-test cdrouter_basic_1
-
Skip the same test using its execution number
$ cdrouter-cli -skip-test 1
Information for all of the cdrouter-cli
options can be displayed at any
point using the help
option:
$ cdrouter-cli -help
/usr/cdrouter/bin/cdrouter-cli [option [value]] ...
-info Display test suite and host information
-silent Display limited login and verification checkpoints
-trace Display all protocol stack log output
-include-proto Include the protocol module in all trace messages
-hide x,y,z Filter out trace messages for specific protocols
-show x,y,z Only show trace messages for specific protocols
-protocols List all registered protocols available for filters
-pt Show one line summary for all packets sent and received
-decode Display decodes of all packets
-dump Display hex dumps of all packets
-config x Set the config file name
-new-config Print new config file and exit
-upgrade-config Print upgraded -config file and exit
-logfile x Specify an output logfile
-print-skipped Print list of skipped tests based on config file and exit
-check-config Check the config file for errors
-skip-log Create logfile skip.log containing reasons for skipped tests
-logdir x Specify an output log directory
-force Automatically remove any existing log directory when starting
-number-format Create test log names based on the test execution number
-capture Automatically generate capture files
-testvar x=y Specify a testvar variable on the commandline to set
-delete-testvar x Specify a testvar variable on the commandline to delete
-testvar_group group:x=y Specify a testvar group variable on the commandline to set
-delete-testvar_group group:x Specify a testvar group variable on the commandline to delete
-create-group group Specify a testvar group on the commandline to create (i.e. remove IGNORE)
-delete-group group Specify a testvar group on the commandline to delete (i.e. add IGNORE)
-show-vars Show all test variables as they are set
-addon x Specify a specific addon to execute
-module x Specify a specific module to execute
-execute x Execute a specific test name/number
-skip-test x Skip execution of a specific test number
-skip-module x Skip execution of a test module
-repeat Repeat all tests forever (see -until-fail)
-rcount x Repeat all tests this many times (in a loop)
-scount x Repeat each test this many times before proceeding
-retry x Retry up to x times if test fails
-rdelay x Add x second delay before retrying a failed test
-shuffle Run the list of test cases in a random order
-seed x Set the seed used with the -shuffle option
-until-fail Stop once a test failure has occurred
-max-fail x Stop when the failure count reaches x
-begin x Begin executing tests from a specific test name/number
-end x Stop executing tests at a specific test name/number
-stop-after x Stop executing after running x tests (x >=0)
-duration sec Run tests for x seconds and then stop
-duration-interrupt Interrupt currently executing test when -duration timeout elapses
-duration-no-error Do not consider -duration timeout elapsing an error
-pause Prompt the user before starting each test
-delay x Add x second delay before each test
-list List test counts for each module
-doc Print out documentation for the test case(s)
-web Print out documentation in web format (HTML)
-csv Print out documentation in CSV format
-xml Print out documentation in XML format
-summary Print out test case summaries only
-xterm Display the test and module as the xterm title
-xterm-stats Display the running count of tests as the xterm title
-color Display pass/fail results in color
-testpath Set the path to search for module files
-no-control-c Display the Control-C handler
-show-system-id Display the System ID
-update-license Updates the QA Cafe license from qacafe.com
-check-version Checks for new version from qacafe.com
-help Display this help
-version Display the script version
Reporting options:
-report-html <file> Generate an HTML report in <file>
-report-template <file> Generate a report based on a template in <file>
-template <file> Specify the name of a template file
-comment <note> Add a comment line to the log and HTML report
Example:
cdrouter-cli -module nat.tcl -trace -xterm-stats -rcount 10