CDRouter Support

CDRouter Performance User Guide

user-guide version 10.4

Introduction

The CDRouter Performance add-on enhances traditional CDRouter functional testing by adding basic performance testing capabilities over IPv4 and IPv6 connections to CDRouter. The CDRouter Performance add-on includes a number of both application throughput and latency test cases which are designed to measure the performance of application level traffic through a DUT. These tests can be combined with existing CDRouter functional tests to track the DUT’s behavior and performance over time.

CDRouter Performance enables:

  • Download, upload, and bi-directional throughput tests for TCP and UDP traffic
    • Control over transmit bandwidth rate and datagram size
    • Control over the number of concurrent streams
  • UDP, DNS query, and DHCP(renew and restart) latency tests
  • Support for both IPv4 and IPv6 connections
  • Support for wired 10 / 100 / 1000 Ethernet and 802.11 a/b/g/n/ac* wireless interfaces
  • Visualization tools that enable graphing of performance data over time

*802.11ac only available with the NTA1000v3 or newer platforms. Full 802.11ac speeds currently supported in the downstream direction only for the NTA1000v3 and NTA1000v4 platforms. The NTA1000v5 platform introduced a new 802.11ac 3x3 wireless card that removes this limitation and adds another transmit/receive antenna.

Licensing

CDRouter Performance is a licensed addon for CDRouter that must be purchased from QA Cafe. For information on upgrading your license to include CDRouter Performance and any other addons, please contact sales@qacafe.com.

To use CDRouter Performance you must install an updated license file from QA Cafe with the CDRouter Performance addon enabled. For help installing CDRouter or an updating a license file, please contact support@qacafe.com.

CDRouter will report the status of all available addons during the installation process and during startup. To verify that CDRouter Performance is enabled on your system, execute the command cdrouter-cli -info and look for the line Performance is enabled, as shown below. If this line is present, CDRouter Performance is enabled and ready to use.

$ cdrouter-cli -info

Starting /usr/cdrouter/bin/cdrouter-cli Mon Aug 14 12:33:54 EDT 2017
Copyright (c) 2001-2017 by QA Cafe
Version 10.4 build 1 (24796 branches/release_10_4), built 2017-08-08 17:49:38 by nightly@cdr-forge6.lan (x86_64)
OS: CentOS Linux 7.3.1611 (4.4.68-1.el7.qacafe.x86_64)
CPU: Intel(R) Core(TM) i7-4790S CPU @ 3.20GHz
Loaded modules from: '/usr/cdrouter/tests'
Start command: /usr/cdrouter/bin/cdrouter-cli -info
Test Suite: cdrouter 10.4.1
System ID: 236e4e779e3f73a21f5c0abaf4db7260
License file: /etc/cdrouter.lic
Serial number: NTA1000-10508
Registered to: qacafe
Maintenance, support and upgrades until: 2018-07-07
Licensed to run: cdrouter
    Multiport   is enabled
    IPv6        is enabled
    Storage     is enabled
    IKE         is enabled
    TR69        is enabled
    TR69-EDM    is enabled
    Nmap        is enabled
    BBF.069     is disabled
    SNMP        is enabled
    Performance is enabled                         <----------------- 
    ICS         is disabled
    DOCSIS      is disabled
NTA platform 5, image 5.2, serial number NTA1000-10508

System Requirements

The CDRouter Performance addon requires CDRouter 9.3 or newer and a supported NTA1000 appliance system from QA Cafe. It includes test cases to measure uni-directional and bi-directional throughput over IPv4. The CDRouter IPv6 addon is also required to test throughput performance over IPv6.

In CDRouter 10.0, two additional LAN-to-LAN Performance modules were added. The CDRouter Multiport add-on is requried to test Performance between two clients on the LAN.

In CDRouter 10.2, Multicast performance tests were added. If your CPE can forward multicast traffic, these modules may be used without the need for another license, besides the ones mentioned above. However, due to the resources required to generate this type of traffic, it will only be supported on the NTA1000v5 appliance hardware.

In CDRouter 10.4, tests to measure DNS and DHCP latency were added to the IPv4 and IPv6 performance modules. No additional license is needed to run these additional test cases.

The NTA1000v5 appliance system supports the CDRouter Performance addon directly, without modification. NTA1000v4, NTA1000v3, and NTA1000v2 appliance systems require additional software and kernel upgrades to support the CDRouter Performance addon. Please see this Knowledge Base article for additional information on upgrading NTA1000 appliance systems to support the CDRouter Performance addon or contact the CDRouter support team support@qacafe.com.

NTA1000v1 and non-NTA1000 systems are not supported to run the CDRouter Performance addon. If you are interested in upgrading an NTA1000v1 system or acquiring a new NTA1000 please contact sales@qacafe.com.

Terminology

CDRouter Performance introduces a few concepts and terms that are defined below:

  • Application layer performance: The measurement of application layer traffic throughput and/or latency through a DUT. Application layer traffic specifically refers to network traffic that simulates data from OSI layers 4 through 7.

  • Application Latency: The measurement of time it takes for an application to make a request and receive a response. Specfically, the amount of time between sending a request and receiving a reply.

  • Bandwidth: The maximum transmit or receive data rate for a particular physical interface, often described in Mb/sec. For example, Fast Ethernet has a bandwidth of 100 Mb/sec whereas Gigabit Ethernet has a bandwidth of 1,000 Mb/sec (1Gb/sec). In the CDRouter Performance addon, bandwidth is a configurable parameter that determines the maximum attempted transmit rate for a particular test.

  • Download test: A performance test that measures a performance metric in the downstream direction from WAN to LAN.

  • DUT: Device under test. This is the CPE or gateway device that is connected to, and being tested by CDRouter.

  • Goodput: This is a measure of actual data that is actually sent or received. This is the raw application data, minus all the frame/packet/segment headers. For example, a 1500-byte frame will have 1472 bytes of actual UDP data within.

  • LAN test: A performance test that measures a performance metric from one LAN client to a second LAN client. This would typically be used to measure either the switching or bridging capability of a DUT.

  • Mb/sec Megabits per second. This is a unit of measure for data transfer speed that is equivalent to one million bits per second. Most data transfer speed metrics in the CDRouter Performance add-on are presented in units of Mb/sec.

  • Retransmits: The total number of retransmitted TCP packets for a given performance test. Retransmits provide a basic measure of loss for TCP connections.

  • Upload test: A performance test that measures a performance metric in the upstream direction from LAN to WAN.

Test Methodology

Overview

Throughput (Goodput) Testing

The CDRouter Performance add-on uses a client-server model for sending and receiving traffic, and measuring performance. When performance is enabled, supportsPerformance set to “yes”, CDRouter will create a performance ‘server’ and a performance ‘client’, one on each interface (2) used in the test. For the LAN-to-WAN performance test cases, the server is defined on the primary WAN interface and the client is on the primary LAN interface. For the LAN-to-LAN performance test cases, the server and client are each defined on their own LAN interface (the primary and a secondary).

The performance server’s IPv4 and IPv6 addresses are specified by the testvars perfWanIp and perfWanIpv6, respectively. The performance client’s IP addresses on the LAN are assigned automatically using either DHCP or static for IPv4, and DHCPv6 or autoconf for IPv6. For the LAN-to-LAN test cases, both, the server and the client are allocated addresses via the above method. The performance client supports being configured on either a wired or wireless interface on the LAN.

When a performance test is run, a control connection is established between the performance client and server. Performance traffic is then transmitted in the upstream or downstream direction by the client or server, respectively. The received traffic is monitored and key performance metrics are then calculated. Bidirectional tests establish a second control connection between the server and client and then performance traffic may be transmitted in both directions (upstream and downstream) simultaneously.

Application Latency Testing

The methodology for application latency testing is similar to throughput testing. The main difference is that the performance server is replaced by a simulated remote host for the UDP latency tests and a simulated DNS server for the DNS latency tests. The DHCP tests are only between a LAN client and the DUT.

In these test cases, the appropriate frame is sent and the time is recorded. When the reply is received, the time is recorded again. The latency is the difference in time between the recorded times. In all cases, several attempts are made and the reported/validated latency is the average of all the times collected.

Performance Metrics

Two key performance metrics, throughput and latency, are currently supported by the CDRouter Performance addon. Loss (UDP) and retries (TCP) are also measured as part of the throughput tests:

  • Throughput: The total amount of network traffic received by the performance endpoint; the performance client in the downstream direction and the performance server in the upstream direction. Throughput is also referred to as ‘achieved bandwidth’ or ‘goodput’ and is measured in Mb/sec. This measurement is calculated using the application payload only and does not include packet headers. In the UDP tests, packets that arrive out of order are still counted when calculating throughput.

    • Loss: For upload tests, it is the difference between the number of UDP packets transmitted by the performance client and the number of UDP packets received by the performance server. For download tests, it is the difference between the number of UDP packets transmitted by the performance server and the number of UDP packets received by the performance client. In either case, UDP packets that arrive out of order are also counted as loss. Loss is expressed as a percentage.

    • Retries: For upload tests, this is the number of times the client had to retry sending the same TCP packet. For download tests, this is the number of times the server had to retry sending the same TCP packet. Retries typically gives an indication of network congestion. In the case of a single device, it may indicate ‘congestion’ on the DUT itself.

  • Latency: For UDP, this is the amount of time it takes for a packet to travel between theperformance client and performance server through the DUT and vice versa. For DHCP and DNS, this is the amount of time between sending a request and receiving a response. Latency is measured in microseconds (usec).

Test Cases

In CDRouter v10.4, the CDRouter Performance addon includes eight (8) test modules containing a total of fifty (50) test cases for verifying basic application layer performance over IPv4 and IPv6:

Test Module Description Number of Tests Other Licenses Required
perf IPv4 performance tests 12
perf-v6 IPv6 performance tests 12 IPv6
perf-lan IPv4 LAN performance tests 9 Multiport
perf-lan-v6 IPV6 LAN performance tests 9 IPv6 and Multiport
perf-mcast IPv4 Multicast performance tests 2
perf-mcast-v6 IPv6 Multicast performance tests 2 IPv6
perf-mcast-lan IPv4 LAN Multicast performance tests 2 Multiport
perf-mcast-lan-v6 IPv6 LAN Multicast performance tests 2 IPv6 and Multiport

Each performance test within these modules verifies a single performance metric in the upstream or downstream directions independently, or in both directions simultaneously (bidirectional), according to the following table:

Test Case Measured Performance Metric Direction Traffic Type
perf_1, ipv6_perf_1, perf_lan_1, ipv6_perf_lan_1 Throughput Downstream TCP
perf_2, ipv6_perf_2, perf_lan_2, ipv6_perf_lan_2 Throughput Upstream TCP
perf_3, ipv6_perf_3, perf_lan_3, ipv6_perf_lan_3 Throughput Downstream UDP
perf_4, ipv6_perf_4, perf_lan_4, ipv6_perf_lan_4 Throughput Upstream UDP
perf_mcast_1, ipv6_perf_mcast_1, perf_mcast_lan_1, ipv6_perf_mcast_lan_1 Throughput Downstream UDP
perf_mcast_2, ipv6_perf_mcast_2, perf_mcast_lan_2, ipv6_perf_mcast_lan_2 Throughput Upsteam UDP
perf_5, ipv6_perf_5, perf_lan_5, ipv6_perf_lan_5 Throughput Both TCP
perf_6, ipv6_perr_6, perf_lan_6, ipv6_perf_lan_6 Throughput Both UDP
perf_7, ipv6_perf_7, perf_lan_7, ipv6_perf_lan_7 Throughout Download UDP
perf_8, ipv6_perf_8, perf_lan_8, ipv6_perf_lan_8 Throughput Upload UDP
perf_9, ipv6_perf_9, perf_lan_9, ipv6_perf_lan_9 Latency Both, one direction at a time UDP
perf_10, perf_11 Latency Both DHCPv4
ipv6_perf_10, ipv6_perf_11 Latency Both DHCPv6
perf_12, ipv6_perf_12 Latency Both DNS

Like all CDRouter test cases, each performance test generates a high-level PASS or FAIL result. In addition, they produce one or more numeric values for the metric being tested. PASS or FAIL results are determined based upon configurable maximum and minimum thresholds for each type of metric, making it easy to quickly identify whether or not the DUT’s performance is consistent over time and meets expected user defined performance requirements.

Test Setup

CDRouter Performance has no additional test setup requirements. Any closed loop test setup used for CDRouter functional testing can also be used for performance testing.

The basic test setup for CDRouter LAN-to-WAN Performance throughtput tests is shown below:

The multiport test setup for CDRouter LAN-to-LAN Performance throughput tests is shown below:

The DNS latency tst setup for CDRouter is shown below:

The DHCP latency tests setups for CDROuter is shown below:

Supported WAN Modes

The CDRouter Performance add-on currently supports the following WAN modes:

  • IPv4 - static, DHCP, PPPoE
  • IPv6 - static, autoconf, DHCPv6, DHCPv6 w/PD, PPPoE

Support for additional WAN modes will be added in future CDRouter releases. Please contact support@qacafe.com for details on specific WAN modes.

Firewalls

The CDRouter Performance addon is firewall-friendly. For all download tests the performance client on the LAN will automatically connect to the performance server on the WAN to establish any firewall or NAT entries before sending performance traffic.

Basic Performance Configuration

The CDRouter Performance add-on includes several testvars that can be used to configure the test environment and the behavior of performance related tests.

To globally enable the Performance add-on for a specific configuration, the testvar supportsPerformance must be set to a value of “yes”. All additional testvars provided with the Performance add-on are defined below.

Basic Configuration

Performance Settings

WAN Download Thresholds

WAN Upload Thresholds

LAN to LAN Thresholds

Application Thresholds

Advanced Options

Note: perfEnableCapture should only be used when additional debugging information is needed to help troubleshoot certain issues, like connectivity issues. For more information, see the Packet Capture and Performance for more information.

Wireless

Please see the following section for more information.

Wireless Configuration

An additional wireless configuration section may be added to the base configuration. This allows a single configuration file to contain different settings for both wired and wireless interfaces. CDRouter will then automatically use the appropriate wired or wireless performance settings based on the underlying interface used for a particular test.

Wireless specific performance settings can be enabled by removing the keyword IGNORE from the wireless-performance testvar_group declaration in the default configuration file.

Note that only the testvars listed below can be defined for both wired and wireless interfaces.

testvar_group wireless-performance {

    SECTION "Performance Settings" {

        testvar perfDownloadBandwidth
        testvar perfUploadBandwidth
        testvar perfLantoLanBandwidth

    }

    SECTION "WAN Download Thresholds" {

        testvar perfDownloadMaxLatency
        testvar perfDownloadTcpRetr
        testvar perfDownloadTcpLowThreshold
        testvar perfDownloadTcpHighThreshold
        testvar perfDownloadUdpLossPercentage
        testvar perfDownloadUdpLowThreshold
        testvar perfDownloadUdpHighThreshold

    }

    SECTION "WAN Upload Thresholds" {

        testvar perfUploadMaxLatency
        testvar perfUploadTcpRetr
        testvar perfUploadTcpLowThreshold
        testvar perfUploadTcpHighThreshold
        testvar perfUploadUdpLossPercentage
        testvar perfUploadUdpLowThreshold
        testvar perfUploadUdpHighThreshold

    }

    SECTION "LAN to LAN Thresholds" {

        testvar perfLantoLanMaxLatency
        testvar perfLantoLanTcpRetr
        testvar perfLantoLanTcpLowThreshold
        testvar perfLantoLanTcpHighThreshold
        testvar perfLantoLanUdpLossPercentage
        testvar perfLantoLanUdpLowThreshold
        testvar perfLantoLanUdpHighThreshold

    }

    SECTION "Application Thresolds" {

        testvar perfDHCPRenewMaxLatency
        testvar perfDHCPRestartMaxLatency
        testvar perfDNSMaxLatency

    }

}

Running Performance Tests

Performance support can be enabled within CDRouter by uncommenting the testvar supportsPerformance and setting it to a value of “yes” within a CDRouter configuration file. Configuration of IPv6 and/or additional LAN interfaces (Multiport) are also required to enable IPv6 performance or LAN-to-LAN performance, respectively.

Performance test cases are selected and added to a test package, the same way that functional test cases are added to a test package.

Once a test package is defined with performance test cases, it may be launched.

Throughput Test Cases

When a throughput performance test is executed, CDRouter will automatically generate and log performance data at an interval specified by the testvar perfInterval. By default, CDRouter will determine an appropriate interval based on the total duration of the test. You can also give this testvar the value ‘0’, which means do not print out interval stats. In this case, only the end-of-test summary is printed in the log. This may be useful when testing with more than 1 stream, specified by perfStreams.

The CDRouter Performance addon allows up to 500 streams to be created between the client and server endpoints. There is also the concept of incrementing the number of streams by specified value perfStreamIncr with each loop interation of the test package.

Here is an example log file showing how performance data is displayed while a throughput performance test is running:

2017-08-03 14:42:14.993 SECTION(cdrouter-4671): Setting up LAN and WAN performance test
2017-08-03 14:42:15.616 INFO(cdrouter-4671): Capture files will filter out performance data

2017-08-03 14:42:17.621 SECTION(cdrouter-4671): Starting performance test server at 202.254.1.4
2017-08-03 14:42:17.621 INFO(cdrouter-4671): Target download bandwidth is 500.0M (0=unlimited)

2017-08-03 14:42:18.628 SECTION(cdrouter-4671): Starting IPv4 UDP performance test at 202.254.1.11
2017-08-03 14:42:18.628 INFO(lan): 
2017-08-03 14:42:18.628 INFO(lan): Connect to performance server at 202.254.1.4:19750 using the following:
2017-08-03 14:42:18.628 INFO(lan): 
2017-08-03 14:42:18.629 INFO(lan):       direction: download ( source 202.254.1.4:19750 -> destination 202.254.1.11 )
2017-08-03 14:42:18.629 INFO(lan):       bandwidth: 500.0M (0=unlimited)
2017-08-03 14:42:18.629 INFO(lan):        duration: 10
2017-08-03 14:42:18.629 INFO(lan):        protocol: UDP
2017-08-03 14:42:18.629 INFO(lan):         streams: 1
2017-08-03 14:42:18.629 INFO(lan):          length: 1472
2017-08-03 14:42:18.629 INFO(lan): 
2017-08-03 14:42:18.655 I<<<(lan-1): b0:75:0c:85:44:41                       b0:75:0c:1b:17:e5                       ARP       202.254.1.4 is at b0:75:0c:85:44:41
2017-08-03 14:42:18.656 INFO(lan): Connecting to host 202.254.1.4, port 19750
2017-08-03 14:42:18.656 INFO(lan): Reverse mode, remote host 202.254.1.4 is sending
2017-08-03 14:42:18.657 I<<<(wan-1): b0:75:0c:1b:17:e5                       ff:ff:ff:ff:ff:ff                       ARP       Who is 202.254.1.4, tell 202.254.1.11
2017-08-03 14:42:18.657 INFO(wan): ARP request, who is 202.254.1.4, tell 202.254.1.11
2017-08-03 14:42:19.733 INFO(lan): [  5] local 202.254.1.11 port 36844 connected to 202.254.1.4 port 19750
2017-08-03 14:42:19.733 INFO(lan): [ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
2017-08-03 14:42:19.733 INFO(lan): [  5]   0.00-1.00   sec  59.0 MBytes   495 Mbits/sec  0.020 ms  472/42532 (1.1%)  
2017-08-03 14:42:20.733 INFO(lan): [  5]   1.00-2.00   sec  59.9 MBytes   503 Mbits/sec  0.024 ms  0/42677 (0%)  
2017-08-03 14:42:21.733 INFO(lan): [  5]   2.00-3.00   sec  59.2 MBytes   497 Mbits/sec  0.018 ms  0/42183 (0%)  
2017-08-03 14:42:22.733 INFO(lan): [  5]   3.00-4.00   sec  59.5 MBytes   499 Mbits/sec  0.017 ms  598/42948 (1.4%)  
2017-08-03 14:42:23.733 INFO(lan): [  5]   4.00-5.00   sec  58.9 MBytes   494 Mbits/sec  0.026 ms  0/41965 (0%)  
2017-08-03 14:42:24.733 INFO(lan): [  5]   5.00-6.00   sec  60.2 MBytes   505 Mbits/sec  0.024 ms  0/42910 (0%)  
2017-08-03 14:42:25.733 INFO(lan): [  5]   6.00-7.00   sec  59.0 MBytes   495 Mbits/sec  0.014 ms  0/42019 (0%)  
2017-08-03 14:42:26.733 INFO(lan): [  5]   7.00-8.00   sec  59.3 MBytes   497 Mbits/sec  0.020 ms  299/42534 (0.7%)  
2017-08-03 14:42:27.733 INFO(lan): [  5]   8.00-9.00   sec  60.1 MBytes   504 Mbits/sec  0.015 ms  0/42835 (0%)  
2017-08-03 14:42:28.813 INFO(lan): [  5]   9.00-10.00  sec  59.2 MBytes   496 Mbits/sec  0.022 ms  0/42144 (0%)  
2017-08-03 14:42:28.813 INFO(lan): - - - - - - - - - - - - - - - - - - - - - - - - -
2017-08-03 14:42:28.814 INFO(lan): [ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
2017-08-03 14:42:28.814 INFO(lan): [  5]   0.00-10.00  sec   596 MBytes   499 Mbits/sec  0.022 ms  1369/424747 (0.32%)  
2017-08-03 14:42:28.814 INFO(lan): [  5] Sent 424747 datagrams
2017-08-03 14:42:28.815 INFO(lan): 
2017-08-03 14:42:28.832 INFO(cdrouter-4671): UDP download throughput over 10.0 seconds is 499 Mbits/sec
2017-08-03 14:42:28.832 INFO(cdrouter-4671): UDP jitter is 0.022 ms with a loss rate of 0.32230952%
2017-08-03 14:42:28.832 PASS(cdrouter-4671): UDP download throughput of 499 Mbits/sec is greater than UDP low threshold of 10.0 Mbits/sec

CDRouter will also generate a final performance report and log it at the end of the test:

2017-08-03 14:42:28.837 SECTION(cdrouter-4671): Displaying all measured performance intervals

Connecting to host 202.254.1.4, port 19750
Reverse mode, remote host 202.254.1.4 is sending
[  5] local 202.254.1.11 port 36844 connected to 202.254.1.4 port 19750
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  59.0 MBytes   495 Mbits/sec  0.020 ms  472/42532 (1.1%)  
[  5]   1.00-2.00   sec  59.9 MBytes   503 Mbits/sec  0.024 ms  0/42677 (0%)  
[  5]   2.00-3.00   sec  59.2 MBytes   497 Mbits/sec  0.018 ms  0/42183 (0%)  
[  5]   3.00-4.00   sec  59.5 MBytes   499 Mbits/sec  0.017 ms  598/42948 (1.4%)  
[  5]   4.00-5.00   sec  58.9 MBytes   494 Mbits/sec  0.026 ms  0/41965 (0%)  
[  5]   5.00-6.00   sec  60.2 MBytes   505 Mbits/sec  0.024 ms  0/42910 (0%)  
[  5]   6.00-7.00   sec  59.0 MBytes   495 Mbits/sec  0.014 ms  0/42019 (0%)  
[  5]   7.00-8.00   sec  59.3 MBytes   497 Mbits/sec  0.020 ms  299/42534 (0.7%)  
[  5]   8.00-9.00   sec  60.1 MBytes   504 Mbits/sec  0.015 ms  0/42835 (0%)  
[  5]   9.00-10.00  sec  59.2 MBytes   496 Mbits/sec  0.022 ms  0/42144 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   596 MBytes   499 Mbits/sec  0.022 ms  1369/424747 (0.32%)  
[  5] Sent 424747 datagrams

Application Latency Test Cases

When an application latency performance test is executed, CDRouter will automatically generate the necessary application traffic and record a timestamp. A timestamp will also be recorded when the reply is received. It will log the difference between these timestmaps as the latency. It will repeat these steps several times and then also log the calculated average latency. This average will then be checked and validated against the value specified as the acceptable latnecy in the CDRouter configuration file.

Here is an example log file showing how performance data is displayed while a application latency performance test is running:

2017-08-10 15:53:50.191 SECTION(cdrouter-4792): Measuring DNS query latency
2017-08-10 15:53:50.191 INFO(lan): DNS lookup host.domain5566b8e25a4f4.cdrouter.com on DNS server 192.168.1.1:53
2017-08-10 15:53:50.192 O>>>(lan-8): 192.168.1.173                           192.168.1.1                             DNS       Query A host.domain5566b8e25a4f4.cdrouter.com
2017-08-10 15:53:50.192 INFO(lan): Waiting up to 5 seconds for a DNS response packet on port 13402
2017-08-10 15:53:50.193 I<<<(wan-7): 202.254.1.2                             202.254.101.1                           DNS       Query A host.domain5566b8e25a4f4.cdrouter.com
2017-08-10 15:53:50.194 INFO(dns1): 202.254.101.1:53 received UDP from 202.254.1.2:62248
2017-08-10 15:53:50.194 INFO(dns1): Received DNS query for host.domain5566b8e25a4f4.cdrouter.com with type A
2017-08-10 15:53:50.194 INFO(dns1): Found DNS entry for host.domain5566b8e25a4f4.cdrouter.com with type A
2017-08-10 15:53:50.194 INFO(dns1): Found matching record: host.domain5566b8e25a4f4.cdrouter.com - Type: A => 2.2.2.2
2017-08-10 15:53:50.194 INFO(dns1): DNS response sent to DNS query (0x9676) from 202.254.1.2:62248
2017-08-10 15:53:50.195 O>>>(wan-8): 202.254.101.1                           202.254.1.2                             DNS       host.domain5566b8e25a4f4.cdrouter.com is at 2.2.2.2
2017-08-10 15:53:50.196 I<<<(lan-9): 192.168.1.1                             192.168.1.173                           DNS       host.domain5566b8e25a4f4.cdrouter.com is at 2.2.2.2
2017-08-10 15:53:50.196 INFO(lan): 192.168.1.173:13402 received UDP from 192.168.1.1:53
2017-08-10 15:53:50.196 INFO(lan): Found UDP session at port 13402
2017-08-10 15:53:50.197 INFO(lan): DNS response with host.domain5566b8e25a4f4.cdrouter.com at 2.2.2.2
2017-08-10 15:53:50.197 PASS(cdrouter-4792): DNS query completed in 3404 usec
2017-08-10 15:53:50.297 PASS(cdrouter-4792): Average DNS latency of 3210 usec is below max latency of 50000

Analyzing Throughput Performance Results

CDRouter measures application level throughput. This is the rate of data passed through TCP or UDP. This rate calculation is independent of the lower level protocol headers which are not included in CDRouter’s throughput measurement. Line-rate is often debated when discussing bandwidth and throughput. CDRouter Performance metrics are based on the application-layer and not at the interface-layer. This is considered ‘usable data line-rate’.

NOTE: Be careful when comparing throughput performance numbers from one test tool to another. It is important to know how the performance results are measured and presented. Some tools may generate performance metrics based on raw packet data rates whereas other tools may base performance calculations on payload or application data rates (such as CDRouter).

CDRouter may be configured with a target bandwidth or an unbounded bandwidth. In the unbounded case, CDRouter will attempt to send traffic as fast as possible for either TCP or UDP. In the case of UDP, unbounded bandwidth may produce high packet loss especially when the forwarding path contains interfaces with different speeds (.i.e to/from a wireless LAN client and a host on the wired WAN).

NOTE: When analyzing UDP throughput results, it’s important to note that packets arriving out of order are counted as a lost packet, but they are still used to calculate acheived bandwidth (throughput). For more information on this topic refer to the Knowledge Base article here.

Understanding the Theoretical Maximum

The theoretical maximum application throughput varies slightly between UDP and TCP. However, the approximate maximum value for each transport protocol can be calculated by subtracting the total length of the lower level headers. For example, a DUT with a Gigabit Ethernet interface on both its WAN and LAN will not achieve a throughput of 1,000Mb/sec. The theoretical maximum application throughput in this case is a bit less due to frame/packet/segment overhead.

The following equations explain how to calculate the theoretical maximum application throughput:

First, for UDP over IPv4, assume that a maximum size Ethernet frame is 1,518 bytes. In order to transmit this frame, an additional 20 bytes must be included for the preamble, frame delimiter and inter-frame gap. This means for each maximum size Ethernet frame sent, a total 1,538 bytes need to be transmitted.

The amount of actual data, or payload, in that frame (without VLAN ID) is:

1,518 -  18 (Ethernet frame) - 20 (IPv4 header) - 8 (UDP header) = 1,472 bytes

So for each maximum sized Ethernet frame that is transmitted, only 1,472 bytes are payload or application data. As a result, the efficiency can be calculated as:

Total Payload in Bytes / Total Number of Bytes Transmitted = Efficiency

In this example, the UDP efficiency is:

1,472 / 1,538 = .957 or 95.7% = UDP Efficiency

Therefore the theoretical maximum application throughput is:

Efficiency * Physical Layer Net Bit Rate = Theoretical Maximum Application Throughput (in bits)

Since Gigabit Ethernet’s physical layer net bit rate is 1,000 Mb/sec, the theoretical maximum application throughput in this example is:

.957 * 1,000 = 957 Mb/sec = Maximum Application Throughput for UDP

For TCP over IPv4, the TCP header is 20 bytes instead of 8. This changes the efficiency calculation, slightly:

1,460 / 1,538 = .949 or 94.9% = TCP Efficiency

As a result, the theoretical maximum application throughput for TCP is less than that of UDP:

.949 * 1,000 = 949 Mb/sec = Maximum Application Throughput for TCP

In practice, the maximum application throughput for TCP is even less than the theoretical maximum because of the need for acknowledgments and due to TCP window settings. A more realistic theoretical maximum application throughput figure for TCP is approximately 940 Mb/sec.

The above calculations were derived from the information presented here.

NOTE: CDRouter uses an IP layer MTU of 1500 bytes on Ethernet for performance testing. It is possible to achieve higher efficiency using jumbo Ethernet frames. Jumbo frames minimize the per packet overhead and may produce a higher application throughput measurement. However, jumbo frames are not supported on many WAN connections.

Implications of PPPoE

PPPoE requires an additional 8 bytes of header information. As a result, the UDP and TCP efficiency decreases when PPPoE is enabled on the WAN.

The UDP efficiency with PPPoE enabled is:

1,464 / 1,538 = .952 or 95.2% = UDP Efficiency

This results in a theoretical maximum application throughput for UDP of:

.952 * 1,000 = 952 Mb/sec = Maximum Application Throughput for UDP

Likewise, the TCP efficiency with PPPoE enabled is:

1,452 / 1,538 = .944 or 94.4% = TCP Efficiency

Which results in a theoretical maximum application throughput for TCP of:

.944 * 1,000 = 944 Mb/sec = Maximum Application Throughput for TCP

Implications of VLANs

Like PPPoE, VLANs impact the UDP and TCP efficiency due to the additional 4 bytes of header that are required.

The UDP efficiency with VLANs enabled is:

1,468 / 1,538 = .954 or 95.4% = UDP Efficiency

This results in a theoretical maximum application throughput for UDP of:

.954 * 1,000 = 954 Mb/sec = Maximum Application Throughput for UDP

Likewise, the TCP efficiency with VLANs enabled is:

1,456 / 1,538 = .947 or 94.7% = TCP Efficiency

Which results in a theoretical maximum application throughput for TCP of:

.947 * 1,000 = 947 Mb/sec = Maximum Application Throughput for TCP

Implications of IPv6

IPv6 uses 40-byte headers instead of 20-byte headers in IPv4. As a result, the UDP and TCP efficiency decreases when IPv6 is used.

1,518 -  18 (Ethernet frame) - 40 (IPv6 header) - 8 (UDP header) = 1,472 bytes

The UDP efficiency for IPv6:

1,452 / 1,538 = .944 or 94.4% = UDP Efficiency

This results in a theoretical maximum application throughput for UDP of:

.944 * 1,000 = 944 Mb/sec = Maximum Application Throughput for UDP

Likewise, the TCP efficiency over IPv6 is:

1,440 / 1,538 = .936 or 93.6% = TCP Efficiency

Which results in a theoretical maximum application throughput for TCP of:

.936 * 1,000 = 936 Mb/sec = Maximum Application Throughput for TCP

And these value are further impacted when PPPoE or VLANs are used with IPv6.

Understanding the Theoretical Maximum for Wireless Interfaces

Determining the theoretical maximum application throughput for a wireless interface is much more complex due to the number of variables involved. Many of these variables are determined by the configuration of the DUT’s integrated access point, including:

  • The band used: 2.4 GHz or 5.0 GHz
  • The channel width: 20, 40, or 80 MHz
  • The 802.11 mode supported: 802.11 a/b/g/n/ac
  • The number of spatial streams supported: 1, 2, or 3

Other variables are environment specific and include the quality of the signal and the amount of interference or noise that is present. Environmental variables can be unpredictable and must be taken into consideration when reviewing performance results generated with wireless interfaces.

Hardware versions 2, 3, 4, and 5 of the NTA1000 platform are supported by the CDRouter Performance add-on. One of the major differences between these hardware versions is the type of wireless interface(s) installed:

NTA1000 Version QCA 802.11 a/b/g/n Adapter Intel 802.11 a/b/g/n/ac Adapter QCA 802.11 a/b/g/n/ac Adapter
v2 Yes No No
v3 No Yes No
v4 Yes Yes No
v5 Yes No Yes

The QCA based adapter found in the NTA1000v2, v4, and v5 platforms is a dual- band 3x3 design that supports 802.11 a/b/g/n. Likewise, the Intel based adapter found in the NTA1000v3 and NTA1000v4 platforms is a dual-band 2x2 design that supports 802.11 a/b/g/n/ac. The NTA1000v5 replaced the Intel adapter with a 2nd QCA 802.11 ac capable 3x3 design. The maximum physical layer data rates supported by these adapters for each band is shown below:

Wirelesss Adapter Band Maximum Physical Layer Data Rate
QCA (N) 2.4 GHz 450 Mbps
QCA (N) 5 GHz 450 Mbps
Intel (AC) 2.4 GHz 300 Mbps
Intel (AC) 5 GHz 867 Mbps
QCA (AC) 2.4 GHz 450 Mbps
QCA (AC) 5.0 GHz 1.27 Gbps

These maximum data rates represent connections in ideal environments with the maximum channel width and all spatial streams enabled.

In practice, every wireless frame that is transmitted must be acknowledged, which has a significant impact on the maximum application throughput that can be achieved. In addition the UDP and TCP efficiency must be factored in which results in maximum application throughput figures that are often no more than 50 to 60% of the maximum physical layer data rates presented above:

Wirelesss Adapter Band Approximate Maximum Application Throughput
QCA (N) 2.4 GHz 225 to 270 Mbps
QCA (N) 5 GHz 225 to 270 Mbps
Intel (AC) 2.4 GHz 150 to 180 Mbps
Intel (AC) 5 GHz 434 to 520 Mbps
QCA (AC) 2.4 GHz 225 to 270 Mbps
QCA (AC) 5.0 GHz 635 to 762 Mbps

It is important to note that these are approximate values that pertain to the capabilities of the NTA1000 platform from QA Cafe. These values should only be used as a guideline. Results will vary from one test setup to another and are highly sensitive to the both the wireless environment and the device configuration.

NOTE: The 802.11 Intel ac adapter installed in the NTA1000 (v3 and v4) currently supports maximum 802.11 ac physical layer data rates in the downstream direction (WAN -> LAN) only. Data rates in the upstream direction (LAN -> WAN) are currently limited by the driver and firmware installed on those NTA1000 platforms.

Pass/Fail Criteria

Each performance test generates a PASS or FAIL test result based on configurable ranges for each performance metric that is measured. If the measured performance for a particular metric falls outside of the acceptable threshold(s), CDRouter will fail the test. Likewise, if the measured performance falls within the acceptable threshold(s), CDRouter will pass the test.

The thresholds for each performance metric can be found in WAN Download Thresholds, WAN Upload Thresholds, LAN to LAN Thresholds, and Application Thresholds sections within the CDRouter Performance Add-On portion of the CDRouter configuration file. Default values have been chosen for all thresholds.

Performance Graphs

One of the features of the CDRouter Performance add-on is the ability to generate graphs of device performance. These graphs make it very easy to verify that the DUT’s performance for key metrics is consistent over time when measured across multiple iterations of a test run. Discrepancies between key performance metrics and performance degradation due to functional testing included in the test run will also be easy to identify.

Applicable Test Cases

CDRouter automatically collects performance data for specific tests in the perf, perf-mcast, perf-v6, perf-v6-mcast, perf-lan, perf-mcast-lan, perf-lan-v6 and perf-mcast-lan-v6 test modules, as listed in the tables below.

Graph Support for IPv4 Performance Tests
Test Case Description Graph Support?
perf_1 IPv4 TCP download throughput test WAN to LAN Yes
perf_2 IPv4 TCP upload throughput test LAN to WAN Yes
perf_3 IPv4 UDP download throughput test WAN to LAN Yes
perf_4 IPv4 UDP upload throughput test LAN to WAN Yes
perf_5 IPv4 TCP bidirectional throughput test No
perf_6 IPv4 UDP bidirectional throughput test No
perf_7 IPv4 UDP download loss threshold test WAN to LAN No
perf_8 IPv4 UDP upload loss threshold test LAN to WAN No
perf_9 IPv4 bidirectional latency test Yes
perf_10 IPv4 DHCP renew latency test Yes
perf_11 IPv4 DHCP restart latency test Yes
perf_12 IPv4 DNS query latency test Yes
perf_mcast_1 IPv4 Multicast UDP download throughput test WAN to LAN Yes
perf_mcast_2 IPv4 Multicast UDP upload throughput test LAN to WAN Yes
perf_lan_1 IPv4 TCP ‘download’ throughput test LAN to LAN Yes
perf_lan_2 IPv4 TCP ‘upload’ throughput test LAN to LAN Yes
perf_lan_3 IPv4 UDP ‘download’ throughput test LAN to LAN Yes
perf_lan_4 IPv4 UDP ‘upload’ throughput test LAN to LAN Yes
perf_lan_5 IPv4 TCP bidirectional throughput test LAN to LAN No
perf_lan_6 IPv4 UDP bidirectional throughput test LAN to LAN No
perf_lan_7 IPv4 UDP ‘download’ loss threshold test LAN to LAN No
perf_lan_8 IPv4 UDP ‘upload’ loss threshold test LAN to LAN No
perf_lan_9 IPv4 bidirectional latency test LAN to LAN No
perf_mcast_lan_1 IPv4 UDP Multicast ‘download’ throughput test LAN to LAN Yes
perf_mcast_lan_2 IPv4 UDP Multicast ‘upload’ throughput test LAN to LAN Yes
Graph Support for IPv6 Performance Tests
Test Case Description Graph Support?
ipv6_perf_1 IPv6 TCP download throughput test WAN to LAN Yes
ipv6_perf_2 IPv6 TCP upload throughput test LAN to WAN Yes
ipv6_perf_3 IPv6 UDP download throughput test WAN to LAN Yes
ipv6_perf_4 IPv6 UDP upload throughput test LAN to WAN Yes
ipv6_perf_5 IPv6 TCP bidirectional throughput test No
ipv6_perf_6 IPv6 UDP bidirectional throughput test No
ipv6_perf_7 IPv6 UDP download loss threshold test WAN to LAN No
ipv6_perf_8 IPv6 UDP upload loss threshold test LAN to WAN No
ipv6_perf_9 IPv6 bidirectional latency test Yes
ipv6_perf_10 IPv6 DHCPv6 renew latency test Yes
ipv6_perf_11 IPv6 DHCPv6 restart latency test Yes
ipv6_perf_12 IPv6 DNS query latency test Yes
ipv6_perf_mcast_1 IPv6 Multicast UDP download throughput test WAN to LAN Yes
ipv6_perf_mcast_2 IPv6 Multicast UDP upload throughput test LAN to WAN Yes
ipv6_perf_lan_1 IPv6 TCP ‘download’ throughput test LAN to LAN Yes
ipv6_perf_lan_2 IPv6 TCP ‘upload’ throughput test LAN to LAN Yes
ipv6_perf_lan_3 IPv6 UDP ‘download’ throughput test LAN to LAN Yes
ipv6_perf_lan_4 IPv6 UDP ‘upload’ throughput test LAN to LAN Yes
ipv6_perf_lan_5 IPv6 TCP bidirectional throughput test LAN to LAN No
ipv6_perf_lan_6 IPv6 UDP bidirectional throughput test LAN to LAN No
ipv6_perf_lan_7 IPv6 UDP ‘download’ loss threshold test LAN to LAN No
ipv6_perf_lan_8 IPv6 UDP ‘upload’ loss threshold test LAN to LAN No
ipv6_perf_lan_9 IPv6 bidirectional latency test LAN to LAN No
ipv6_perf_mcast_lan_1 IPv6 Multicast UDP ‘download’ throughput test LAN to LAN Yes
ipv6_perf_mcast_lan_2 IPv6 Multicast UDP ‘upload’ throughput test LAN to LAN Yes

Creating Graphs

Every time a supported test case is is executed CDRouter will log the final performance number (throughput or latency) that was achieved by the DUT. These data points can then be graphed using either the Visualize Performance or the Visualize Latency features included with CDRouter’s Visualize tool.

All of the data points for each supported performance test case within a single test run are plotted as a separate time series on the performance graph. As a result, the best way to build a meaningful performance graph is to utilize CDRouter’s looping and repeating features to ensure that multiple data points are collected for each performance test case within the same test run.

Example Graphs

Performance graphs are interactive, and all available performance data for the selected test run is displayed by default. Individual series can be removed from the graph by clicking on them in the legend. Likewise, clicking on an individual data point within the graph will automatically load the log file for that particular test case.

Here is an example performance graph showing UDP and TCP throughput for IPv4.

This graph shows significant degradation for all throughput metrics over time. These results were achieved by mixing performance throughput and functional tests in the same test run. It should be noted that this same device did not exhibit performance degradation unless certain functional tests are included in the test run.

This graph shows the IPv4 application latency for UDP, DHCP and DNS.

Performance Testing Considerations

TR-069

CDRouter’s TCP offload interface should be used for the ACS in any configurations where both TR-069 and performance are enabled. The TCP offload interface is enabled by default, and can be explicitly enabled by configuring the acsUseTcpOffload to a value of “yes”.

testvar acsUseTcpOffload yes

Disparate Network Interface Speeds

Packet loss may occur when testing a DUT with different network interface speeds on the WAN and LAN. This is especially true for UDP performance tests since UDP does not have any ability to adapt to available bandwidth, unlike TCP.

Some common examples are testing download speeds from a Gigabit Ethernet WAN to slower 802.11 wifi LAN interfaces. If realistic bandwidth targets are not set, CDRouter will send UDP traffic at gigabit speeds on the WAN, which will result in significant packet loss on the LAN. Likewise, DSL and DOCSIS gateways may have slower WAN speeds and therefore suffer from the same problem during upload tests.

To combat this issue, reasonable bandwidth targets for both upstream and downstream testing should be set using the testvars perfUploadBandwidth and perfDownloadBandwidth, respectively.

Note that some experimentation may be required to determine acceptable bandwidth targets for a particular test setup and configuration.

Testing Exercises

Analyze Performance Over Time

A great testing exercise is to monitor the performance of a DUT over time by creating a test package with a mix of performance and functional test cases and looping it.

Looping the test package will generate meaningful performance graphs that highlight the DUT’s performance over time. Adding functional tests to the package make it possible to verify whether or not various protocol interactions have an impact on the performance of the DUT.

Looping performance tests alone is a good way to generate baseline measurements. Any discrepancies between TCP and UDP, IPv4 and IPv6, or upload and download performance will be easy to spot.

Validate QoS Settings

CDRouter Performance makes it possible to validate QoS settings if the DUT has the ability to throttle data traffic based on IP address, transport protocol, or port number.

Many of the performance metrics measured by CDRouter have associated “high” thresholds that can be configured. Enabling and setting these thresholds based on specific QoS policies in the DUT make it possible to verify that the DUT’s QoS policy is actually working as expected.

Mix Performance and Functional Tests

Performance tests can be added to any existing CDRouter test package. Running performance tests alone is a good way to generate baseline measurements. Mixing performance and functional tests together and looping the test package is a great way to identify performance degradation over time due to protocol interactions. DHCP, DNS, NAT, and many other protocols can have an impact on performance. With CDRouter Performance it is easy to identify these trends.

Our testing has identified a few interesting performance issues using this technique. In one case a DUT’s performance was stable and consistent when only performance tests were run. However, when a few simple functional tests were added to the same package, significant performance degradation was witnessed after only a few hours of testing. This particular DUT seemed to be especially sensitive to DHCP client interactions on its WAN interface.

Packet Capture and Performance

CDRouter disables packet captures for performance traffic by default. Performance traffic is not displayed in the main log file or interface capture files. It is possible to enable capture files for performance traffic by setting the testvar perfEnableCapture to “yes”.

testvar perfEnableCapture yes

Enabling performance captures also has a slight impact on the performance capabilities of the NTA1000 and will produce large capture files that are not easily viewed. A short performance duration is recommended if capture files are enabled. Alternatively, the perfEnableCapture may be set to a value of “control” which will filter and capture only TCP session setup and ICMP traffic. The control capture option greatly reduces the overall size of the capture file making it suitable for viewing within CDRouter’s web interface. Control traffic is useful in some cases for debugging performance tests.

Note that performance results with captures enabled may be difficult to export from CDRouter’s web interface since the capture files are part of the export. It is also possible to quickly fill the disk on the system by running with capture files enabled for an extended period of time.

For example, a single 10 second performance test with packet captures disabled uses approximately 130KB of disk space. The same 10 second test with control capture enabled used 135KB. However, the same 10 second test with full capture enabled used 315MB.

Limitations

  • As mentioned in the Test Setup section, CDRouter Performance currently supports a limited number of WAN modes. Specifically static, DHCP and PPPoE for IPv4, and static, autoconf, DHCPv6, DHCPv6 w/PD, and PPPoE for IPv6.

  • CDRouter Performance is able to drive line rate bandwidth on a 1 Gigabit Ethernet interface using 1500 byte IPv4/v6 packets. For UDP traffic, the total bandwidth generated by the NTA1000 (models v2/v3/v4) will drop if small packets/frames are configured. The NTA1000 can not drive line rate 1 Gigabit bandwidth using 64 byte packets.

  • For bi-directional testing, the total bandwidth of the system is approximately 1.7 Gb/sec. Future NTA1000 platforms will address this overall throughput limitation.

  • The 802.11ac adapter installed in the NTA1000v3 and NTA1000v4 currently supports maximum physical layer data rates in the downstream direction (WAN to LAN) only. Data rates in the upstream direction are currently limited by the driver and firmware available for the adapter and installed on the NTA1000. The 802.11ac adapter installed in the NTA1000v5 does not have this limitation. (The NTA1000v2 platform does not have 802.11ac adapter installed.)

Contents

×

About CDRouter

CDRouter is made by QA Cafe, a technology company based in Portsmouth, NH.

Get in touch via our Contact page or by following us on your favorite service: