Parallel Testing: Running Multiple Packages at Once

Overview

CDRouter can run multiple test packages in parallel. The number of simultaneous test runs is limited by the number of available test instances and physical interfaces (see Licensing below). Running packages in parallel is supported for hardware versions NTA1000v5 and above.

After clicking the Launch button for a package (or launching it via the API), CDRouter will do one of three things:

  • Run the package, if no other packages are currently running.
  • Run the package, if the interfaces used by the package do not overlap with those of any currently running package and if there is an available test instance.
  • Add the package to the Job Queue, if the package uses interfaces that overlap with any currently running packages or if there are no more available test instances.

When a test instance becomes available and there are packages in the Job Queue, CDRouter will select the next package that uses interfaces that do not overlap with any currently running package. Therefore, the Job Queue does not necessarily reflect the order in which packages will be run.

Scheduling packages using the package scheduler will result in the above behavior at the time the package is scheduled to start.

If a CDRouter system includes only one test instance, packages will be run sequentially in the order they are launched or scheduled.

Licensing

Running packages in parallel requires additional test instances beyond the one (1) that’s included by default with every CDRouter system. Additional test instances must be purchased and installed on the system, up to a maximum of 5 test instances per system.

See the QA Cafe license FAQ for more information about test instances and how to purchase additional ones.

Requirements for Running Packages in Parallel

The single requirement that determines whether packages run in parallel or get queued is physical interfaces: two packages can only run at the same time if their CDRouter configurations don’t use any of the same physical interfaces on the CDRouter system (see Considerations below for a few exceptions involving ICS and VLANs).

In practice, setting this up takes three steps:

  1. Connect each DUT to its own physical interfaces. Give each device a dedicated WAN and LAN (and any other interfaces its testing requires) on the CDRouter system. No two DUTs should share a physical port.
  2. Create a separate CDRouter configuration for each DUT, with each configuration referencing only the interfaces connected to that DUT.
  3. Launch a package against each configuration. As long as an available test instance exists (see Licensing above) and the configurations don’t share physical interfaces, CDRouter runs the packages at the same time instead of adding them to the Job Queue.

Basic Parallel Test Setup

The diagram below shows the simplest version of this setup: two DUTs, each connected directly to its own dedicated WAN and LAN interfaces on the CDRouter system, so their packages can run in parallel:

image

Advanced Parallel Test Setup

Some test environments require WAN traffic to pass through an access concentrator, such as an OLT or CMTS, before reaching each DUT. You can still give each DUT its own dedicated CDRouter interfaces in this case by separating their WAN traffic onto different VLANs. The diagram below shows an example of this setup for a GPON environment using an OLT:

image

Why Use Parallel Testing?

Running packages in parallel lets a CDRouter system do more testing in less time by using multiple test instances at once. There are several common reasons to take advantage of this:

  • Reduce test cycle time. Split a large package (or test suite) into several smaller packages, and run them at the same time against separate physical DUTs of the same make, model, and firmware version. Instead of running everything sequentially against a single DUT, tests finish in a fraction of the total time.

  • Test multiple hardware or firmware versions at once. Connect two or more DUTs to the CDRouter system and run the same package against each simultaneously. These DUTs can be different hardware revisions, firmware builds, or units of the same model. This is useful when validating a new firmware release across hardware variants, or comparing results between builds, without waiting for each test run to finish before starting the next.

  • Expand test coverage. Since more testing completes in the same amount of wall-clock time, teams can add packages, test cases, or configurations to a regular test cycle without lengthening how long that cycle takes to run.

  • Add stability testing alongside conformance testing. Run a long-duration stability or soak package on one set of test instances while continuing day-to-day conformance and interoperability testing on others, so long-running tests don’t block or delay routine testing.

Each of these approaches requires enough test instances (see Licensing above) and physical interfaces (see Requirements for Running Packages in Parallel above) for the number of packages you want running at once.

Considerations

There are a few limitations to CDRouter’s parallel testing feature:

  • Multiple packages using an ICS interface may be run in parallel, but they must use non-overlapping WAN ISP networks. See the IPv4 and IPv6 configuration examples below.

  • Multiple packages using the same ICS interface may be run in parallel, so long as all of their non-ICS interfaces are available.

  • CDRouter uses physical interfaces when determining if interfaces overlap. Therefore when using VLANs, devices must use unique physical ports on the system in order to run testing in parallel.

  • If your test environment uses a routed setup, such as Scenario A or Scenario B described in Connecting CDRouter to a GPON Network, the Reserved Address Block and WAN networks used by each configuration you want to run in parallel must all be unique. This is because a static route pointing to CDRouter’s WAN IP must be added on either the access concentrator (OLT, CMTS, etc.) or an intermediate router in the lab network. Overlapping networks between parallel configurations would create conflicting routes, since a router can only route a given network to one next-hop at a time.

When planning for high-volume testing using multiple packages running in parallel, it is important to consider if any package might block other packages from running for the reasons listed above. Those packages should be queued last or separately in order for parallel tests to finish running on all available test instances. Packages that could cause inadvertent blockages include packages that use multiport, wireless, ICS, or VLANs.

IPv4 Configuration Example for ICS

To configure non-overlapping WAN ISP networks, give each configuration that shares an ICS interface its own unique wanIspIp, wanIspAssignIp, and wanIspMask testvars, so that CDRouter’s simulated ISP network for each DUT doesn’t overlap with any other. For example, the configuration for DUT 1 might set:

testvar wanIspIp                         202.254.1.1
testvar wanIspAssignIp                   202.254.1.2
testvar wanIspMask                       255.255.255.0

while the configuration for DUT 2, sharing the same ICS interface, uses a different subnet:

testvar wanIspIp                         202.254.2.1
testvar wanIspAssignIp                   202.254.2.2
testvar wanIspMask                       255.255.255.0

IPv6 Configuration Example for ICS

IPv6 has a similar set of constraints: each configuration also needs its own non-overlapping WAN network (ipv6WanIspIp, ipv6WanIspAssignIp, and ipv6WanIspPrefixLen) and delegated LAN prefix (dhcpv6WanAssignPrefix and dhcpv6WanAssignPrefixLen). For example, the configuration for DUT 1 might set:

testvar ipv6WanIspIp                     3001::1
testvar ipv6WanIspAssignIp               3001::2
testvar ipv6WanIspPrefixLen              64
testvar dhcpv6WanAssignPrefix            3001:dddd::
testvar dhcpv6WanAssignPrefixLen         48

while the configuration for DUT 2, sharing the same ICS interface, uses different values:

testvar ipv6WanIspIp                     3002::1
testvar ipv6WanIspAssignIp               3002::2
testvar ipv6WanIspPrefixLen              64
testvar dhcpv6WanAssignPrefix            3002:dddd::
testvar dhcpv6WanAssignPrefixLen         48