CDRouter Speed Test User Guide

Introduction

The CDRouter Speed Test expansion adds IP Layer Capacity testing to CDRouter based on Broadband Forum’s TR-471. TR-471 defines a standardized methodology for measuring IP Layer Capacity using UDP-based throughput tests that are initiated and controlled through a device management protocol.

Unlike traditional performance testing where CDRouter controls both traffic endpoints, Speed Test works by instructing the DUT to perform a speed test against a CDRouter-hosted test server. The test is triggered via a management protocol (CWMP or USP), and the DUT reports the measured IP Layer Capacity back to CDRouter. This approach validates the DUT’s own speed test implementation as defined by the TR-471 specification.

CDRouter Speed Test enables:

  • TR-471 IP Layer Capacity testing using search algorithms B and C
  • Fixed rate IP Layer Capacity testing from 100 Mbps to 10 Gbps in 100 Mbps increments
  • Support for both CWMP (TR-069) and USP (TR-369) management protocols
  • Support for both IPv4 and IPv6
  • Receiver (download) and sender (upload) test directions
  • Configurable test sub-intervals and pass/fail thresholds
  • Performance graphing of throughput metric

Licensing

CDRouter Speed Test is a licensed expansion that must be purchased from QA Cafe prior to use. For information on upgrading a license to include CDRouter Speed Test or any other expansions, please contact sales@qacafe.com.

System Requirements

The CDRouter Speed Test expansion requires an NTA1000v7 or newer system from QA Cafe.

Required Expansion

At least one of the following management protocol expansions is required to run the Speed Test expansion:

  • TR-069 – Required for CWMP-based test modules
  • USP – Required for USP-based test modules

Optional Expansion

  • IPv6 – Required for IPv6 test modules

Terminology

Term Description
IP Layer Capacity A measurement of the maximum IP layer throughput achievable between two endpoints, as defined in TR-471. This is measured in megabits per second (Mbps).
Search Algorithm B A search algorithm that adjusts the sending rate to converge on the maximum IP Layer Capacity of the network path.
Search Algorithm C An improved search algorithm that adjusts the sending rate to converge on the maximum IP Layer Capacity of the network path.
Fixed Rate Test An IP Layer Capacity test performed at a specific, predetermined sending rate. The test verifies that the DUT achieves at least 95% of the target rate.
Receiver Test A test in which the DUT acts as the receiver (download direction). CDRouter’s test server sends UDP traffic to the DUT while the DUT measures and reports the achieved throughput.
Sender Test A test in which the DUT acts as the sender (upload direction). The DUT sends UDP traffic to CDRouter’s test server while both endpoints measure throughput.
Sub-Interval A discrete measurement period within a single test run. Multiple sub-intervals are used and the results are averaged to produce the final IP Layer Capacity summary.
IPLayerCapacitySummary The average throughput in Mbps across all sub-intervals of a test, as reported by the DUT.
LossRatioSummary The average packet loss percentage across all sub-intervals of a test.
RTTRangeSummary The range of round trip time measurements across all sub-intervals of a test.
OB-UDPST Open Broadband UDP Speed Test, the reference implementation of the TR-471 test protocol used by CDRouter’s test server.

Test Methodology

Overview

CDRouter Speed Test uses the TR-471 IP Layer Capacity diagnostic to measure the DUT’s throughput capabilities. The test methodology works as follows:

  1. CDRouter starts an OB-UDPST test server on the WAN interface using the configured speedTestWanIp or speedTestWanIpv6 address.
  2. CDRouter uses a management protocol (CWMP or USP) to trigger the Device.IP.Diagnostics.IPLayerCapacity() diagnostic on the DUT.
  3. The DUT performs the speed test against CDRouter’s test server, sending or receiving UDP traffic depending on whether the test role is sender or receiver.
  4. Upon completion, the DUT reports the results back to CDRouter via the management protocol.
  5. CDRouter validates the reported IP Layer Capacity against the configured threshold and compares the DUT’s reported metrics with server-side measurements.

Search Algorithm Tests vs Fixed Rate Tests

CDRouter Speed Test includes two categories of tests:

  • Search algorithm tests: These tests use the TR-471 rate adjustment algorithms to dynamically search for the DUT’s maximum IP Layer Capacity. The pass/fail threshold is configured by the user via the tr471ReceiverLowThreshold and tr471SenderLowThreshold testvars.

  • Fixed rate tests: These tests send traffic at a specific fixed rate and verify that the DUT achieves at least 95% of the target throughput. Fixed rate tests are available for rates from 100 Mbps to 10 Gbps in 100 Mbps increments. This makes it possible to verify multiple target rates in a single test run by including the appropriate fixed rate tests.

The strategy and requirements of the user will determine which tests make the most sense.

Search algorithm tests are useful for discovering the maximum IP Layer Capacity of a DUT without prior knowledge of its capabilities. The algorithms will automatically adjust the sending rate to converge on the maximum throughput.

Fixed rate tests require minimal configuration and the results are easy to interpret. For any given test, the rate, direction, address family, and management protocol are encoded in the test name. A passing result indicates the DUT achieved at least 95% of the target rate. A failing result indicates it did not.

Test Setup

CDRouter Speed Test has no additional hardware requirements beyond the standard CDRouter test setup. The test setup is the same as for CWMP testing or USP testing, depending on which management protocol is used.

A USP-enabled DUT must support the Device.IP.Diagnostics.IPLayerCapacity() command. A CWMP-enabled DUT must support the Device.IP.Diagnostics.IPLayerCapacityMetrics. object. The management protocol (CWMP or USP) must be configured and functional before running Speed Test test cases.

Supported WAN Modes

CDRouter Speed Test supports the following WAN modes:

  • IPv4 – static, DHCP
  • IPv6 – static, autoconf, DHCPv6, DHCPv6 with PD

Support for additional WAN modes may be added in future CDRouter releases.

Performance Metrics

Each Speed Test test case measures and reports the following metrics:

  • IPLayerCapacitySummary – The average IP Layer throughput in Mbps as reported by the DUT, compared with the server-side measurement.
  • LossRatioSummary – The average packet loss percentage across all test sub-intervals.
  • RTTRangeSummary – The round-trip time range as reported by the DUT (when available).

Test Modules

The CDRouter Speed Test expansion includes 8 test modules:

Test Module Description Number of Tests Other Licenses Required
tr471-cwmp TR-471 IPv4 search algorithm tests using CWMP 4 TR-069
tr471-cwmp-v6 TR-471 IPv6 search algorithm tests using CWMP 4 TR-069 and IPv6
tr471-usp TR-471 IPv4 search algorithm tests using USP 4 USP
tr471-usp-v6 TR-471 IPv6 search algorithm tests using USP 4 USP and IPv6
tr471-fixed-cwmp TR-471 IPv4 fixed rate tests using CWMP (100 Mbps to 10 Gbps) 200 TR-069
tr471-fixed-cwmp-v6 TR-471 IPv6 fixed rate tests using CWMP (100 Mbps to 10 Gbps) 200 TR-069 and IPv6
tr471-fixed-usp TR-471 IPv4 fixed rate tests using USP (100 Mbps to 10 Gbps) 200 USP
tr471-fixed-usp-v6 TR-471 IPv6 fixed rate tests using USP (100 Mbps to 10 Gbps) 200 USP and IPv6

Search Algorithm Test Cases

Each search algorithm module contains 4 tests covering both algorithms (B and C) and both test directions (receiver and sender):

Test Case Algorithm Direction Description
tr471_ipv4_search_b_receiver_protocol B Download Binary search for maximum IP Layer Capacity as receiver
tr471_ipv4_search_b_sender_protocol B Upload Binary search for maximum IP Layer Capacity as sender
tr471_ipv4_search_c_receiver_protocol C Download Exponential search for maximum IP Layer Capacity as receiver
tr471_ipv4_search_c_sender_protocol C Upload Exponential search for maximum IP Layer Capacity as sender

The IPv6 modules follow the same pattern with tr471_ipv6_ prefixed test names.

Fixed Rate Test Cases

Each fixed rate module contains 200 tests: 100 receiver tests and 100 sender tests. The tests cover rates from 100 Mbps to 10,000 Mbps (10 Gbps) in 100 Mbps increments. Each test has a pass/fail threshold set to 95% of the target rate.

Test names follow the pattern:

  • tr471_ipv4_fixed_receiver_protocol_rate – Download test at specified rate
  • tr471_ipv4_fixed_sender_protocol_rate – Upload test at specified rate

For example, tr471_ipv4_fixed_receiver_cwmp_500 is a 500 Mbps download test using CWMP over IPv4. The DUT must achieve at least 475 Mbps (95% of 500) for the test to pass.

Configuration

Enabling Speed Test

To enable the Speed Test expansion, set the supportsSpeedTest testvar to yes:

testvar supportsSpeedTest yes

Speed Test also requires a management protocol to be configured. Enable either CWMP or USP (or both) depending on which test modules will be used:

For CWMP-based tests:

testvar supportsCWMP yes

For USP-based tests:

testvar supportsUSP yes

Refer to the CDRouter TR-069 User Guide or CDRouter USP User Guide for details on configuring the management protocol.

Server Configuration

The following testvars configure the Speed Test server that runs on CDRouter’s WAN interface:

The speedTestWanIp and speedTestWanIpv6 testvars specify the IP addresses of CDRouter’s Speed Test server on the WAN. These addresses must be on the same network as the DUT’s WAN interface and next-hop gateway. By default, these addresses are automatically computed based on the WAN network configuration.

The speedTestPort testvar specifies the UDP port used for the Speed Test control protocol. The default port is 25000.

TR-471 Test Settings

The following testvars control the behavior of the TR-471 IP Layer Capacity tests:

The tr471NumberTestSubIntervals testvar specifies the number of measurement sub-intervals per test. Each sub-interval produces an independent throughput measurement, and all sub-intervals are averaged to compute the final IP Layer Capacity summary. The default is 10 sub-intervals.

The tr471TestSubInterval testvar specifies the duration of each sub-interval in milliseconds. The default is 1000 ms (1 second). With the default settings, each test runs for approximately 10 seconds of active measurement.

The tr471ReceiverLowThreshold and tr471SenderLowThreshold testvars specify the minimum acceptable throughput in Mbps for receiver (download) and sender (upload) search algorithm tests, respectively. If the DUT’s reported IP Layer Capacity is above the threshold, the test passes. If it is below, the test fails. The default threshold for both is 10.0 Mbps.

Note: These threshold testvars apply only to the search algorithm tests. Fixed rate tests use a built-in threshold of 95% of the target rate.

Example Configurations

Basic IPv4 Configuration with CWMP

testvar supportsSpeedTest    yes
testvar supportsCWMP         yes
testvar speedTestWanIp       202.254.1.4
testvar speedTestPort        25000

Basic IPv4 Configuration with USP

testvar supportsSpeedTest    yes
testvar supportsUSP          yes
testvar speedTestWanIp       202.254.1.4
testvar speedTestPort        25000

IPv6 Configuration with USP

testvar supportsSpeedTest    yes
testvar supportsUSP          yes
testvar supportsIPv6         yes
testvar speedTestWanIpv6     3001::4
testvar speedTestPort        25000

Custom Test Duration and Sub-Intervals

testvar tr471NumberTestSubIntervals  20
testvar tr471TestSubInterval         2000

This configuration runs each search algorithm test with 20 sub-intervals of 2 seconds each (40 seconds of active measurement).

Custom Thresholds

testvar tr471ReceiverLowThreshold    100.0
testvar tr471SenderLowThreshold      50.0

This configuration sets the receiver threshold to 100 Mbps and the sender threshold to 50 Mbps.

Possible Problems

  • Speed Test tests are skipped – Verify that supportsSpeedTest is set to yes and that the Speed Test expansion is enabled in your CDRouter license. Run cdrouter-cli -info to confirm.

  • CWMP tests fail to trigger the diagnostic – Ensure that the DUT supports the Device.IP.Diagnostics.IPLayerCapacityMetrics data model object and that CWMP is configured correctly. Refer to the CDRouter TR-069 User Guide for CWMP configuration details.

  • USP tests fail to trigger the diagnostic – Ensure that the DUT supports the Device.IP.Diagnostics.IPLayerCapacity() command and that USP is configured correctly. Refer to the CDRouter USP User Guide for USP configuration details.

  • Tests fail with throughput below threshold – For search algorithm tests, adjust the tr471ReceiverLowThreshold or tr471SenderLowThreshold testvars to match the expected capabilities of the DUT. For fixed rate tests, select test cases with rates appropriate for the DUT’s link speed.

  • Server IP address conflicts – The speedTestWanIp and speedTestWanIpv6 addresses must be free (unused) IP addresses on the WAN network. If the default computed addresses conflict with other devices on the network, set them explicitly in the configuration.

  • Client and server throughput values do not match – CDRouter compares the DUT’s reported IP Layer Capacity with the server-side measurement. Small discrepancies are normal due to measurement timing differences. A test will log a failure if the values diverge significantly, which may indicate a problem with the DUT’s measurement implementation.

  • OBUDPST Version – CDRouter currently utilizes obudpst v8.2.0 which works both the v8.1.0 and v8.0.0. obudpst v7.X clients will not connect with the server. For more information about this, please read this obudpst issue