BYO Software Regression Testing on Rescale

Mark Whitney

 regressiontesting2
Rescale is a valuable regression and performance testing resource for software vendors. Using our API and command line tools, we will discuss how you can run all or a subset of your in-house regression tests on Rescale. The advantages of testing on Rescale are as follows:

  1. Compute resources are on-demand, so you only pay when you are actually running tests
  2. Compute resources are also scalable, enabling you to run a large test suite in parallel and get feedback sooner
  3. Heterogeneous resources are available to test software performance on various hardware configurations (e.g. Infiniband, 10GigE, Tesla and Phi coprocessors)
  4. Testing your software on Rescale can then optionally enable you to provide private beta releases to other customers on Rescale

Test Setup

For the remainder of this article, we will assume you have the following sets of files:

  1. Full build tree of your software package, in a commonly supported archive format (tar.gz, zip, etc.)
  2. Archived set of reference test inputs and expected outputs
  3. Script or command line to run your software build against one or more test inputs
  4. Script to evaluate actual test output with expected output
  5. (Optional) Smaller set of incremental build products to overlay on top of the full build tree

In the examples below, we will be using our python SDK. A selection of examples below are available in the SDK repo here. The SDK just wraps our REST API, so you can port these examples to other languages by using the endpoints referenced in https://github.com/rescale/python-sdk/blob/master/rescale/client.py.
Note that all these examples require:

  1. An account on the Rescale platform
  2. A local RESCALE_API_KEY environment variable set to your API key found in Settings->API from the main platform page

Running tests from a single job

We will start with the simplest example, uploading a full build and test reference data as job input files, running the tests serially, and comparing the results. Let’s start with some example “software” which we will upload and run. Here is a list of the software package and test files:

echoware/bin/echo.sh (just echos test in to test out)
test[0-9]/in (test input)
test[0-9]/expected_out (expected test output)
test[0-9]/out (actual test run output)

Each software build and test case is archived separately. Here are the steps to prepare and run our test suite job:

  1. Upload build, reference test data, and results comparison script using the Rescale python SDK:
#!/usr/bin/env python3
import rescale.client
TEST_ARCHIVE = 'inputs/all_tests.tar.gz'
BUILD_ARCHIVE = 'inputs/echoware0.1.tar.gz'
POST_COMPARE_SCRIPT = 'inputs/compare_results.sh'
input_files = [rescale.client.RescaleFile(file_path=TEST_ARCHIVE),
               rescale.client.RescaleFile(file_path=BUILD_ARCHIVE),
              ]
post_process_file = \
    rescale.client.RescaleFile(file_path=POST_COMPARE_SCRIPT)

RescaleFile uploads the local file contents to Rescale and returns metadata to reference that file. At this point, you can view these files at  https://www.rescale.com/route/files/.
      2. Create the test suite job:

TEST_COMMAND = """
for testcase in $(find . -name "test[0-9]*" -type d); do
    ./echoware/bin/echo.sh $testcase
done
"""
POST_RUN_COMPARE_COMMAND = """
for testcase in $(find . -name "test[0-9]*" -type d); do
    ./compare_results.sh $testcase
done
"""
JOB_NAME = 'echoware0.1-all-tests'
job_definition = {
    'name': JOB_NAME,
    'isLowPriority': True,
    'jobanalyses': [
       {
           'analysis': {
               'code': 'custom'
           },
           'hardware': {
                'coresPerSlot': 1,
                'slots': 1,
                'coreType': {
                   'code': 'standard-plus'
              }
           },
            'inputFiles': [{'id': inp.id} for inp in input_files],
            'command': TEST_COMMAND,
            'postProcessScript': {'id': post_process_file.id},
            'postProcessScriptCommand': POST_RUN_COMPARE_COMMAND
        }
    ],
}
job = rescale.client.RescaleJob(json_data=job_definition)

RescaleJob creates a new job which you can now view at https://www.rescale.com/route/jobs/. Note here we are running the job on a single Marble core. You can opt to run more cores by increasing coresPerSlot or change the core type by selecting a different core type code from RescaleConnect.get_core_types().
Note that the command and postProcessScriptCommand fields can be any valid bash script, so there is quite a bit of flexibility in how you run your test and evaluate the results. In our very simple example, the post-test command comparison just diffs the out and expected_out files in each test case directory.

  1. Submit the job for execution and wait for it to complete:
job.submit()
job.wait()

Once the job cluster is provisioned, the input files are transferred to the cluster, unencrypted, then uncompressed in the work directory. Next, the TEST_COMMAND is run, followed by the POST_RUN_COMPARE_COMMAND.

  1. Download the test results. All Rescale job commands have stdout redirected to process_output.log let’s just download that one file to get the test result summary.
STDOUT_LOG = 'process_output.log'
test_log = job.get_file(STDOUT_LOG)
test_log.download(target=test_log.name)

It is important to note here that by doing our test result comparison as a post-processing step in our job, we avoid downloading potentially large output files, which would delay how long it takes to get test results. This doesn’t address the issue that we still need to upload the test reference outputs, which will often be similar size to the actual output. The key is that we only need to upload files to Rescale when they change, not for every test job we launch. Assuming our reference test cases do not change very often, we can now reuse the files we just uploaded to Rescale in later test runs.
You can find this example in full here.

Reusing reference test data

We will now modify the above procedure to avoid uploading reference test data for every submitted test job.

  1. (modified) Find test file metadata on Rescale and use as input file to subsequent jobs:
# omitting var setup from above
original_test_file = rescale.client.RescaleFile.get_newest_by_name(TEST_ARCHIVE)
input_files = [original_test_file,
               rescale.client.RescaleFile(file_path=BUILD_ARCHIVE),
              ]
post_process_file = \
    rescale.client.RescaleFile.get_newest_by_name(POST_COMPARE_SCRIPT)

RescaleFile.get_newest_by_name just retrieves metadata for the test file that was already uploaded to Rescale. Note that if you uploaded multiple test archives with the same name, this will pick the most recently uploaded one.
Steps 2 through 4 are the same as the previous example.

Parallelize long running tests

The previous examples just run all your tests sequentially, let’s now run some in parallel. For this example, we assume our tests are partitioned into “short” and “long” tests. The short tests are in an archive called all_short_tests.tar.gz and each long test is in a separate archive called long_test_.tar.gz.
We will now launch a single job for all the short tests and a job per test for the long tests. We assume these test files have already been uploaded to Rescale, as was done in the first example.

# omitting command var setup
SHORT_TEST_ARCHIVE = 'inputs/all_short_tests.tar.gz'
LONG_TEST_FORMAT = 'inputs/long_test_{i}.tar.gz'
LONG_TEST_COUNT = 10
BUILD_ARCHIVE = 'inputs/echoware0.1.tar.gz'
POST_COMPARE_SCRIPT = 'inputs/compare_results.sh'
# find by name on Rescale instead of uploading from local copy
short_test_bundle = \
    rescale.client.RescaleFile.get_newest_by_name(SHORT_TEST_ARCHIVE)
long_test_inputs = [
    rescale.client.RescaleFile.get_newest_by_name(LONG_TEST_FORMAT.format(i=i))
for i in range(LONG_TEST_COUNT):
    post_process_file = \
      rescale.client.RescaleFile.get_newest_by_name(POST_COMPARE_SCRIPT)
# upload local copy
build_input = rescale.client.RescaleFile(file_path=BUILD_ARCHIVE)
def create_job(name, test_input, core_type, core_count):
    input_files = [build_input, test_input]
    job_definition = {
        'name': name,
        'isLowPriority': True,
        'jobanalyses': [
            {
               'analysis': {
                    'code': 'custom'
                },
               'hardware': {
                    'coresPerSlot': core_count,
                    'slots': 1,
                    'coreType': {
                        'code': core_type
                    }
                },
                'inputFiles': [{'id': inp.id} for inp in input_files],
                'command': TEST_COMMAND,
                'postProcessScript': {'id': post_process_file.id},
                'postProcessScriptCommand': POST_RUN_COMPARE_COMMAND
            }
        ],
    }
    return rescale.client.RescaleJob(json_data=job_definition)
# create all test jobs
short_test_job = create_job('echoware0.1-all-short-tests',
                            short_test_bundle,
                            'standard-plus',
                            1)
long_test_jobs = [create_job('echoware0.1-long-test-{0}'.format(i),
                             long_test,
                            'hpc-plus',
                            32)
                  for i, long_test in enumerate(long_test_inputs)]
test_jobs = [short_test_job] + long_test_jobs
# submit all
[job.submit() for job in test_jobs]
# wait for all to complete
[job.wait() for job in test_jobs]
# get results
[job.get_file(STDOUT_LOG).download(target='{0}.out'.format(job.name))
 for job in test_jobs]

In this example, we launched our short test job with a single Marble core and each of our long tests with a 32 core (2 nodes) Nickel MPI cluster.
This test job configuration is particularly appropriate for performance tests. To test that a particular build + test case combination scales, you might launch 4 jobs with 1, 2, 4, and 8 nodes respectively.
This example can be found here.

Incremental builds

In the above, we avoided re-uploading test data for each test run by reusing the same data already stored on Rescale. If we have a large software build we need to test, we would like to also reuse already uploaded data, but each build tested will generally be different. In many cases though, only a small subset of files from the whole package will change from build to build.
To leverage the similarity in builds, we can supply an incremental build delta that will be uncompressed on top of the base build tree we uploaded in the first job. There are just 2 requirements:

  1. The build delta must have the same directory structure as the base build
  2. We need to specify the build delta archive as an input file AFTER the base build archive
Here is an excerpt with this change:
FULL_BUILD_ARCHIVE = 'inputs/echoware0.1.tar.gz'
BUILD_DELTA = 'inputs/echoware0.2-delta.tar.gz'
# find by name on Rescale
base_build_input = \
    rescale.client.RescaleFile.get_newest_by_name(FULL_BUILD_ARCHIVE)
# upload local copy
incremental_build_input = rescale.client.RescaleFile(file_path=BUILD_DELTA)
def create_job(name, test_input, core_type, core_count):
    input_files = [base_build_input, test_input, incremental_build_input]
    job_definition = {
        'name': JOB_NAME,
        'isLowPriority': True,
        'jobanalyses': [
            {
               'analysis': {
                    'code': 'custom'
                },
                'hardware': {
                    'coresPerSlot': 1,
                    'slots': core_count,
                    'coreType': {
                        'code': core_type
                    }
                },
                'inputFiles': [{'id': inp.id} for inp in input_files],
                'command': TEST_COMMAND,
                'postProcessScript': {'id': post_process_file.id},
                'postProcessScriptCommand': POST_RUN_COMPARE_COMMAND
            }
        ],
    }
    return rescale.client.RescaleJob(json_data=job_definition)

In the above, base_build_input comes from the file already on Rescale and incremental_build_input is uploaded each time.

Parallelism in Design-of-Experiment (DOE) jobs

Another way to run tests is to group multiple tests into a single DOE job. The number of tests that can run in parallel is then defined by the number of task slots you configure for your job. You would then structure your test runs so that they can be parameterized by a templated configuration file, as described in https://www.rescale.com/resources/getting-started/doe/.
This method has the advantage of eliminating job cluster setup time, compared to the multi-job case. The disadvantage is that each test run is limited to the same hardware configuration you define for a task slot. For an example on how to set up a DOE job with the python SDK, see https://github.com/rescale/python-sdk/tree/master/examples/doe.

Large file uploads

In the above examples, we have uploaded our input files with a simple PUT request. This will be slow and/or often not work for multi-gigabyte files. An alternative is to use the Rescale CLI tool, which provides bandwidth optimized file uploads and downloads and can resume transfers if they are interrupted.
For more info on the Rescale CLI, see here: support@rescale.com.
Running tests on Rescale is a great way to reduce testing time and strain on internal compute resources for large regression and performance test suites. The Rescale API provides a very flexible way to launch groups of tests on diverse hardware configurations, including high-memory, high-storage, infiniband, and GPU-enabled clusters. If you are interested in doing your own testing on Rescale, check out our SDK and example scripts at https://github.com/rescale/python-sdk or contact us at support@rescale.com.

Similar Posts