Using CDRouter with Jenkins

Overview

Jenkins is a popular open source continuous integration tool used to manage the workflow of building and testing software. In a CPE firmware build environment, CDRouter can be combined with Jenkins to provide the testing verification of each firmware build.

Jenkins Test Results

Jenkins has built in concepts for testsuites and testcases. It natively recognizes the junit XML test result format. While not all CDRouter test concepts directly map to Jenkins, it is possible to translate CDRouter test results into junit XML test results that Jenkins can understand.

Example Work Flow

Much of the Jenkins process is controlled by scripts that run on the Jenkins server. These scripts handle installing new firmware on target CPE devices and then starting CDRouter to perform testing. This document will not cover the specifics of building and deploying a firmware image since these are vendor specific practices. This document also assumes you are running CDRouter 10.0 or newer.

Using the Web API

CDRouter’s Web API provides full control over executing CDRouter test packages and managing results. QA Cafe has also developed a python module with the API to make interactions with CDRouter even easier.

In most cases, scripts used to drive CDRouter testing will call out to a CDRouter system to start one or more test packages. The script can wait until the tests are complete and then gather the final results. Scripts may also want to export CDRouter test results for a backup or perform other result processing.

Web API Tokens

When the CDRouter user model is enabled, the web API requires the use of an API token. Each user is assigned an API token automatically. It may be beneficial to create a user specifically for Jenkins automation. Please see the Authorization section of web API for more information.

Using Tags

CDRouter’s tag concept can be used to select test packages and organize test results. Results may be tagged with firmware versions, device versions, or other test environment information.

A Simple Example

This example uses a script written in python using the cdrouter.py module available on GitHub. You can find a link to the cdrouter.py repository in the User Guides section of the CDRouter Support website.

The following python script shows how to start a test package, wait for completion, and then convert the results into a format Jenkins can understand. The script only supports CDRouter packages that are fully automatedand run without requiring user interaction.

#!/usr/bin/env python

import sys
import time

from cdrouter import CDRouter
from cdrouter.configs import Testvar
from cdrouter.jobs import Job

if len(sys.argv) < 4:
    print('usage: <base_url> <token> "<package name>"')
    sys.exit(1)

base = sys.argv[1]
token = sys.argv[2]
package_name = sys.argv[3]

# create service
c = CDRouter(base, token=token)


p = c.packages.get_by_name(package_name)
if not p:
    sys.exit("Package {0} not found.".format(package_name))


print('Running package "{0}"'.format(package_name))
print('')

j = c.jobs.launch(Job(package_id=p.id))

# wait for job to be assigned a result ID
while j.result_id is None:
    time.sleep(1)
    j = c.jobs.get(j.id)

print('Test package launched. Job ID: {0}'.format(j.id))
print('Result-ID: {0}'.format(j.result_id))
print('')

print('Waiting for job to complete...')
u = c.results.updates(j.result_id, None)
while u:
    r = c.results.get(j.result_id)
    if r.status in ['completed', 'stopped', 'error']:
        break
    print('Progress: {0}%'.format(u.progress.progress))
    u = c.results.updates(j.result_id, u.id)

result_url = base.strip('/') + '/results/' + str(r.id)
print('Job status: {0}'.format(r.status))
print('')

if r.status != 'completed':
    print('Error: test package "{0}" did not complete successfully ({1})'.format(package_name, r.status))
    print('Test report: {0}'.format(result_url))
    sys.exit(1)


print('Test results:')
print('')
print('{0:>15} : {1}'.format('Summary',      r.result))
print('{0:>15} : {1}'.format('Start',        r.created))
print('{0:>15} : {1}'.format('Duration',     r.duration))
print('{0:>15} : {1}'.format('Package',      r.package_name))
print('{0:>15} : {1}'.format('Config',       r.config_name))
print('{0:>15} : {1}'.format('Tags',         r.tags))
print('{0:>15} : {1}'.format('Pass',         r.passed))
print('{0:>15} : {1}'.format('Fail',         r.fail))
print('')
print('{0:>15} : {1}'.format('Test Report',  result_url))
print('{0:>15} : {1}'.format('Test Summary', result_url + '/print'))
print('')


filename = 'results_{0}.xml'.format(j.result_id)
print('Writing results to file "{0}" in Jenkins XML format'.format(filename))
with open(filename, 'w') as f:
    f.write('<testsuite name="CDRouter" package="{0}" failures="{1}" tests="{2}">\n'.format(package_name, r.fail, r.tests))
    for t in c.tests.iter_list(j.result_id):
        f.write('    <testcase name="{0}">\n'.format(t.name))
        if t.result in ['fail', 'fatal']:
            f.write('        <failure message="test failures">')
            f.write('View the full CDRouter test log {0}'.format(base+'/results/'+str(j.result_id)+'/tests/'+str(t.seq)))
            f.write('</failure>\n')
        elif t.result in ['pending', 'skipped']:
            f.write('        <skipped/>\n')
        f.write('    </testcase>\n')
    f.write('</testsuite>\n')
f.close()


print('done')

This script will produce the following output:

./jenkins.py http://nonpod 6200ab9b "My Test Package"
Running package "My Test Package"

Test package launched. Job ID: 351
Result-ID: 20200905131151

Waiting for job to complete...
Progress: 0%
Progress: 5%
Progress: 27%
Progress: 79%
Progress: 100%
Job status: completed

Test results:

        Summary : The package completed successfully
          Start : 2020-09-05 13:11:51.336049
       Duration : 5
        Package : Generic-Test
         Config : generic_no_DUT_needed_12.3.2
           Tags : [u'generic']
           Pass : 0
           Fail : 0

    Test Report : http://nonpod/results/20200905131151
   Test Summary : http://nonpod/results/20200905131151/print

Writing results to file "results_20200905131151.xml" in Jenkins XML format
done

Additional Notes

The script requires the following arguments in order to run:

The CDRouter base URL (eg. http://nta1000.lan)
The API Token of the user running the test. If Automatic Login is enabled, any dummy token value may be used. See the CDRouter User Model knowledge base article for details on creating users.
The name of the test package to run

When the test package has completed successfully, the results written to a file in the Jenkins JUnit test report XML format.

Previous versions of this article provided examples using a Ruby gem module to facilitate integration with Jenkins. The ruby gem for CDRouter has been discontinued and is no longer supported.

Help Us Improve

The cdrouter.py python module is hosted on GitHub. We welcome push requests and feature suggestions to allow CDRouter to work better with Jenkins and other CI tools.

https://github.com/qacafe/cdrouter.py

You can find a link to the cdrouter.py repository in the User Guides section of the CDRouter Support website.

Product

Search Results