High Throughput Computing (HTC)

Please note that this feature is currently in beta and not enabled by default. Please contact your organization’s Solution Architect or Account Executive to enable access.

Overview

Rescale High Throughput Computing (HTC) is an API-based Rescale product for running a large number (e.g., hundreds to millions) of concurrent small and decoupled jobs (e.g., single-digit cores to single-digit instances and no multi-node MPI).

High Performance Computing (HPC) uses clusters to solve advanced computational tasks. Large tasks are split into subtasks and are solved, in parallel, on a multi-node cluster. An HPC cluster is a set of homogeneous, interconnected machines (nodes). Each node runs a copy of a pre-installed program working on data representing part of a task. To arrive at a global solution, nodes in an HPC cluster need to communicate with each other, and therefore need to be in proximity (same data center, same rack). HPC computations are tightly coupled and fail if individual nodes fail.

The aim of HPC is to reduce time-to-result of an individual computation and maximize operations per second. The lifetime of an on-demand, cloud HPC cluster is as short as possible.

High Throughput Computing (HTC) uses multiple nodes over long periods of time to solve computational tasks. Each job runs a program included in a container image stored in a registry. Similarly to HPC, a global task (batch) is split into subtasks (jobs). These subtasks are loosely coupled. A job, encapsulated by a task, is usually independent of other jobs. The result of an HTC run, often an aggregate of individual results. Sparse communication between jobs allows for distribution of tasks across multiple regions. Thanks to loose coupling, failure of one job does not affect other jobs in a batch and can be restarted.

The aim of HTC is to maximize the number of completed jobs in a week or a month, how many, instead of how fast.

The Rescale HTC API

The following diagram represents all resources exposed by the Rescale HTC API. The subsequent sections describe these in more detail.

Authentication and authorization

The Rescale API key is a non-expiring API key used to access the entire Rescale Platform via the Rescale Public API. You specify the Rescale API in the Authorization Header of an HTTPS request in the following manner to obtain a User Bearer Token:

curl -H “Authorization: Token <Rescale API KEY>” https://htc.rescale.com/api/v1/auth/token 

The User Bearer Token is a temporary Bearer token generated using your permanent API key above and is used for the Rescale HTC API. By default, the temporary bearer token expires in 6 hours. You specify the temporary User Bearer Token in the Authorization Header on an HTTPS request to the HTC API in the following manner:

curl -H “Authorization: Bearer xyz123blablatemp” https://htc.rescale.com/api/v1/htc/projects