Test Labels

Test Labels

This section describes how test labels defined with buddy::label_define in a testpath’s labels.dic and calls to skip_labels in test-constraints properties in a testpath’s config.dic are used to determine which tests are run and which are skipped for a given execution of cdrouter-cli.

labels.dic and buddy::label_define

Labels serve as a way to define a set of test cases and test modules which are skipped when a certain constraint is not met. For example, one might create a label which contains tests which are skipped if IPv6 is not enabled or if the LAN DHCP pool has less than 3 addresses.

Labels are defined in labels.dic with a call to buddy::label_define. Note that the label definition only lists the tests/modules associated with a label, the logic for when to actually skip tests with a given label is done in test-constraints properties in config.dic.

Below is the signature for buddy::label_define:

  buddy::label_define NAME {
      reason "A one-line summary for why the tests got skipped"
  
      description {
  
This text tells the user how to get tests with this label to run.
  
      }
  
      modules {
          module1.tcl
          module2.tcl
          module3.tcl
      }
  
      tests {
          test1
          test2
          test3
      }
  }

The reason string is used as a one-line summary and should concisely explain why the tests/modules were skipped. The description string should be a longer paragraph which explains in more detail why the tests/modules were skipped and helps the user configure the DUT/configuration file so that the tests will run. The modules and tests lists should contain the names of modules and tests which should be skipped.

By convention, label names are all lowercase with words delimited using hyphens (-). For example, a label should be named my-label-name and not my_label_name or myLabelName. Most labels in labels.dic are in the form requires-XXX or incompatible-with-XXX.

Below is an example of a label which defines a set of test cases which should be skipped if the DUT does not have a SIP ALG:

  buddy::label_define requires-sip-alg {
      reason "Skipping SIP ALG related tests"
  
      modules {
          sip-alg.tcl
          sip-alg-tcp.tcl
      }
  }

Note that since there were no test cases associated with the label, the tests list was omitted. The same can be done with the modules list if there are no test modules are associated with a label.

Inheritance

A label can also inherit test/modules from other labels by defining an inherit property. The value of the inherit property is Tcl script which can use the special set command to perform arbitrary set arithmetic on other labels. Use the label command to return the members of an existing label. By default, label returns both test and modules, but an optional second argument of tests or modules can filter its return value to return only tests or only modules.

There exists a special label all which contains all tests and modules which have been loaded. It is not found in labels.dic but is instead created automatically by cdrouter-cli. This label can be used in combination with the inherit property to define a label’s members by specifying the tests or modules that it should not contain.

For example, in the following example the label all-but-tests-a-b-c would skip everything except tests a, b and c:

  buddy::label_define all-but-tests-a-b-c {
      inherit {
          set difference [ label all tests ] { a b c }
      }
  }

or all-but-modules-d-e-f would skip everything except modules d.tcl, e.tcl and f.tcl:

  buddy::label_define all-but-modules-d-e-f {
      inherit {
          set difference [ label all modules ] { d.tcl e.tcl f.tcl }
      }
  }

Additionally, requires-1 would inherit all the members of the label requires-0:

  buddy::label_define requires-1 {
      inherit {
          label requires-0
      }
  }

requires-a-b-c would inherit the union of labels requires-a, requires-b and requires-c:

  buddy::label_define requires-a-b-c {
      inherit {
          set union [ label requires-a ] [ label requires-b ] [ label requires-c ]
      }
  }

and the label requires-a-and-b would inherit the intersection of labels requires-a and requires-b:

  buddy::label_define requires-a-and-b {
      inherit {
          set intersect [ label requires-a ] [ label requires-b ]
      }
  }

Note that a label that defines an inherit property can still define tests and modules lists.

  buddy::label_define requires-a-b-c {
      inherit {
          set union [ label requires-a ] [ label requires-b ] [ label requires-c ]
      }
  
      modules {
          d.tcl
          e.tcl
          f.tcl
      }
  
      tests {
          t1
          t2
          t3
      }
  }

See the TclLIB struct::set documentation for a full list of available set arguments.

config.dic and the test-constraints property

The test-constraints property is assigned to testvars in config.dic which are used in determining whether a set of tests defined by a label should be skipped. The value of this property is Tcl script which is evaluated similarly to the constraints property and thus all the same commands are available (i.e. commands such as this, tv, exists, values, cdrouter_addon, etc.). See Testvar Constraints for more info on testvar constraints and the commands available.

Whereas a call to buddy::label_define in labels.dic defines a label and declares its tests/modules, the test-constraints property inspects the values of relevant testvars and determines if those tests/modules should be skipped. The example below defines a test-constraints property for the testvar supportsSipAlg which skips test/modules with the requires-sip-alg label if supportsSipAlg is no:

buddy::testvar_define supportsSipAlg {
    test-constraints {
        if { [ this value ] == "no" } {
            skip_labels requires-sip-alg
        }
    }
}

Note that the skip_labels call can take an arbitrary number of labels to skip:

skip_labels label-name-1 label-name-2 label-name-3

A call to skip_labels will add all tests/modules in the label’s modules and tests lists to cdrouter-cli’s skiplist, using the reason string from the label definition as the skip reason.

Modifying an existing label

Sometimes it is necessary to add new tests/modules to an existing label. For example, one may want to add an IPv6-related module in a custom testpath to the requires-ipv6 label. This can be done by calling buddy::label_define, passing any new tests/modules which should be added to the existing set assigned to that label:

  buddy::label_define NAME {
      modules {
          additional-module1.tcl
          additional-module2.tcl
          additional-module3.tcl
      }
  
      tests {
          additional-test1
          additional-test2
          additional-test3
      }
  }

Please note that buddy::label_define will throw an error if the label already exists and properties other than tests or modules are given.