The CDRouter Backup and Restore Tools

Introduction

CDRouter is installed with a couple of new command-line tools to help customers backup (and restore) their CDRouter data.

Background

The QA Cafe Support team strongly encourages every customer to do routine backups of their CDRouter data to ensure a successful recovery from some unexpected loss of data or hardware failure.

In the past, the support team provided a Knowledge Base article with a set of instructions as the recommended way to backup your testing data, What’s the best way to backup and restore CDRouter? This article is still relevant today, however…

In an attempt to make this task easier, the CDRouter development team has created both a Backup and Restore tool, specifically for CDRouter’s saved user data.

Starting with CDRouter 10.6, both a Backup (cdrouter-backup) and Restore (cdrouter-restore) now exist as additional command line tools located in /usr/cdrouter/bin.

CDRouter Backup Tool

# ./cdrouter-backup 
Usage: ./cdrouter-backup [OPTION...] LOCATION
Backup the CDRouter data on this system.

LOCATION can be a path on the local file system or a path on a remote
SSH system [[user@][hostname:]][path].

Options:

  -force
    	Do not ask for confirmation before continuing.
  -insecure
    	Allow SSH connections with no host key validation.
  -known-hosts FILE
    	Use SSH known hosts FILE for host key validation.
  -no-compress
    	Create an uncompressed backup.
  -private-keys FILES
    	Use comma-separated SSH private keys FILES for public key authentication.
  -restore
    	Restore from given backup.
  -version
    	print version information and exit.

If the path in LOCATION is a directory, cdrouter-backup will create a
backup named cdrouter-backup-YYYYMMDDHHmmss in that directory.

If -known-hosts is not given, the default is ~/.ssh/known_hosts.

If -private-keys is not given, the default is ~/.ssh/id_dsa,
~/.ssh/id_ecdsa, ~/.ssh/id_ed25519 and ~/.ssh/id_rsa.

The CDRouter Backup tool will gather all the saved user data within the /usr/cdrouter-data directory tree. This includes all the generated results, configuration files, packages, devices and the CDRouter database. By default, it will archive/compress/transfer to the specified destination into a tar/zipped file with the name cdrouter-backup-<YYYYMMDDHHmmss>. The archiving is done locally, but the compression will occur during the transfer to the destination.

Depending on the amount of data saved, this process could potentially take a long time. The actual time it will take is dependent on several factors. Some of these are (but not limited to) the amount of data that is being backed up and the speed of the network link between CDRouter and the remote destination.

The CDRouter software will automatically be stopped when this process is occurring, so no tests may be run while the backup tool is running.

The remote destination you choose should have enough space to store a single large data file. The theoretical max of the data that is being backed up is the size of the data disk in the NTA1000, which is 4TB. This does not mean that your destination needs this much space. The compression will reduce this to a significantly smaller size. (For example, a 227GB database was compressed into a 53GB file)

(Note: The CDRouter Backup tool uses SFTP to securely transfer files. If the destination is a remote system, it must support/enable SFTP.)

Usage Example:

/usr/cdrouter/bin/cdrouter-backup root@172.16.1.211:/cdrouter-backup

When the above command is launched, the CDRouter Backup tool will calculate and display the total size that will be backed up and then will automatically begin the transfer/compression/archive process.

The tool will display statistics as it makes progress. It will display the running total of how much data has been backed up, the percent complete, the speed of the network transfer, and the approximate time remaining to complete the backup.

checking CDRouter system, this may take a while on large systems... done.
backing up 226.82 GiB to /cdrouter-backup/cdrouter-backup-2018031213336 on 172.16.1.211
stopping CDRouter... done.
creating backup... 123.50 GiB / 226.82 GiB [==========================>-----------------------] 54.45% 60.00 MiB/s 27m43s

(Note: as mentioned above, the Backup tool will only backup files in the /usr/cdrouter-data directory tree. If you have other files, such as your own Linux-executable scripts, SSL certificates or custom test cases, located in other directories, they would need to be backed up separately. Of course, we encourage that these additional files are stored under /usr/cdrouter-data so they may be included in backups done with this tool.)

CDRouter Restore Tool

# ./cdrouter-restore
Usage: ./cdrouter-restore [OPTION...] LOCATION
Restore the CDRouter data on this system from a backup.

LOCATION can be a path on the local filesystem or a path on a remote
SSH system [[user@][hostname:]][path].

Options:

  -force
    	Do not ask for confirmation before continuing.
  -insecure
    	Allow SSH connections with no host key validation.
  -known-hosts FILE
    	Use SSH known hosts FILE for host key validation.
  -private-keys FILES
    	Use comma-separated SSH private keys FILES for public key authentication.
  -verify
    	Verify backup before restoring.
  -version
    	print version information and exit.

If -known-hosts is not given, the default is ~/.ssh/known_hosts.

If -private-keys is not given, the default is ~/.ssh/id_dsa,
~/.ssh/id_ecdsa, ~/.ssh/id_ed25519 and ~/.ssh/id_rsa.

The CDRouter Restore tool will access a remote backup file located at the destination specified. It will transfer/extract the data contained within (defined devices, configuration files, packages, and results) into CDRouter’s /usr/cdrouter-data directory on the system where the Restore tool was executed.

Depending on the size of the backup file, this process could take a long time. The actual time it will take is dependent on several factors. Some of these are (but not limited to) the size of the file being restored and the speed of the network link between CDRouter and the remote destination.

The CDRouter software will be stopped when this process is occurring, so no tests may be run while the restore tool is running.

Example:

/usr/cdrouter/bin/cdrouter-restore root@172.16.1.211:/cdrouter-backup/cdrouter-backup-20180312123342

When the above command is launched, the CDRouter Restore tool will calculate and display the total size that will be restored and it will warn the user that any files in the /usr/cdrouter-data directory will be overwritten and will ask if the user wishes to proceed. If the response is ‘yes’, then it will automatically begin the transfer/extraction process.

The tool will display statistics as it makes progress. It will display the running total of how much data has been restored, the percent complete, the speed of the network transfer, and the approximate time remaining to complete the restore.

checking CDRouter system, this may take a while on large systems... done.
restoring from /cdrouter-backup/cdrouter-backup-20180312123342 on 172.16.1.211
WARNING:

* 226.82 GiB will be copied from /cdrouter-backup/cdrouter-backup-20180312123342 to this system.

* This program will erase the CDRouter data directory of the local
  system and replace it with the data from /cdrouter-backup/cdrouter-backup-20180312123342.

* All data that is currently in /usr/cdrouter-data/ will be PERMANENTLY DELETED.  THIS
  ACTION CANNOT BE UNDONE AND MUST NOT BE INTERRUPTED ONCE STARTED.

Are you sure you want to do this (yes/no)? yes

stopping CDRouter... done.

restoring backup... 123.50 GiB / 226.82 GiB [==========================>-----------------------] 54.45% 60.00 MiB/s 27m43s

Getting Help

If you have any questions about either of these command line tools or run into any problems, please contact QA Cafe Customer Support, and we will be happy to help you.