Remote Cluster Access
Rescale has recently released additional user account job settings, which provide customers with the ability to securely connect to the remote clusters that they have provisioned in order to monitor or manage their running simulations more directly.
Users of HPC simulation tools can typically relate to the scenario wherein an analysis model is run for the first time, but the solver crashes for any number of reasons upon initializing the model’s computational domain and beginning the iterative solution. Another situation that users commonly encounter is one whereby a job has been submitted but is missing a requisite input file or an input parameter is set to an inappropriate value. Having the ability to login to the cluster directly enables users to more economically resolve these and similar issues when leveraging Rescale’s HPC platform.
Configuring your account settings on Rescale is straightforward and involves the following four steps:
• Generating an SSH key pair
• Adding the SSH public key to your Rescale account
• Disabling the automatic termination of clusters option in your Rescale account’s job settings
• Logging into a running cluster
Generating an SSH Key Pair
This section highlights for users of both *nix and Windows OS environments how to generate OpenSSH key pairs. If you’re familiar with SSH key generation, or have already generated a key pair previously that you use regularly, then you can skip ahead to the next section.
For users of *nix based operating systems, a command line utility exists named ssh-keygen that can be used to generate these public and private SSH keys from a shell prompt with the following:
ssh-keygen -t rsa -b 2048
Invoking this command will begin generating both public and private keys and prompt you for a location and filename on your local filesystem where you wish to store both keys. Alternatively, you can pass the ssh-keygen command the -f flag along with a path on your local filesystem where you want to store the keys (e.g. /home/rescale/Desktop/id- rsa) as its argument. Using this example, the private key file named id-rsa and the public key file named id- rsa.pub are both saved to subdirectory Desktop of user rescale’s home directory. If you omit the -f flag altogether, then you can opt to use the default which will place both id-rsa and id-rsa.pub files in a subdirectory of your home folder named .ssh (e.g. /home/rescale/.ssh).
You are also prompted to set a passphrase, and again to verify this passphrase, which is then used each time the public key is substantiated with the private key. By default, setting this passphrase is ignored, and doing so is typically preferred to avoid any hassle of entering this passphrase when logging into a remote host. Once the ssh-keygen command line utility has completed, then a 2048 bit RSA key pair should have been generated, and both private and public keys stored in their respective files at the location in your filesystem that was specified.
The options used to create the SSH keys—key type (RSA) and the number of bits created in the key (2048)—happen to correspond to the defaults used when invoking the ssh-keygen utility from a shell command prompt. As a result, equivalent keys could have been generated by just invoking ssh-keygen by itself at the shell prompt. These options have been highlighted here to inform end-users of options they may wish to leverage when generating their own keys. For example, those conscientious about security may wish to generate a 4096 bit key instead of the 2048 bit key used in this example. Note that if users opt for generating DSA keys rather than RSA keys, then they are limited to using exactly 1024 bit keys as specified by FIPS 186-2.
On Windows systems, there are a number of options available for generating an OpenSSH key pair. One popular tool that users typically have available on their systems is PuTTY. If not already installed, it is easy to download and straightforward to install. Once installed, start the puttygen.exe executable. In the section of The PuTTYGen window labeled “Parameters”, click the radio button next to SSH-2 RSA and enter the value 2048 in the field next to “Number of bits in a generated key:” as illustrated in Fig. 2.
Once both parameters have been set as described, then click the button labeled “Generate” in the “Actions” section of the window to begin producing the SSH key pair. Users are next prompted to move their mouse cursor inside of the designated “blank” area within the PuTTYGen window to introduce some randomness into the key generation process. Continue moving the mouse in the designated area of the window until the blue progress bar indicates that key generation has completed. Finally, save both public and private keys to separate files on the local filesystem using the corresponding buttons located in the “Actions” section of the PuTTYGen window.
Once both keys have been saved in their respective files, then users should ensure they safeguard the the private key and note its location on their local filesystem
Adding the SSH Public Key to your Rescale Account
Once an OpenSSH public and private key pair have been generated, users can log in to their Rescale accounts and navigate to the ”Job Settings” page via their user account menu as highlighted in Fig. 3.
Users can open the file that they saved the SSH public key in their preferred text editor, copy its contents, and paste the text into the field labeled “SSH Public Key” on the “Job” settings page. Users may optionally further restrict access to the cluster hosting their simulation by specifying a Classless Inter-Domain Routing (CIDR) rule to limit access to the cluster to either a single IP address or a range of IP addresses depending on a user’s preference. Rules can be easily generated to allow access from “Everywhere”, or “My Current IP Address” using the pull-down menu options available. Users can also create a custom rule by editing the “Access Restriction (CIDR)” field directly.
Disabling Automatic Termination of Clusters
Clusters that host a user’s running simulation are provisioned and configured on-demand. Once the simulation has completed running, the user’s runtime files are moved off of the cluster, and the cluster resources are shut down. If you’re troubleshooting issues with your simulation, you may want to avoid this automatic shut down behavior, so that you can login to the cluster directly, review and correct any issues that arise, and restart the simulation. In order to prevent the cluster from shutting down automatically, uncheck the box next to the option labeled “Terminate Clusters Automatically” at the very bottom of the “Job” settings page.
Note, that users disabling this option are responsible for terminating their clusters manually, and Rescale is not responsible for charges that accrue as a result of user forgetfulness which leaves the cluster running beyond the intended duration. Users should be mindful of the potential repercussions that may result when disabling this option.
Logging Into a Running Cluster on Rescale
Once your public SSH key has been added to your Rescale account’s job settings and you submit a new job, Rescale’s platform uploads the public SSH key associated with your account to the cluster that you provision. The job status message window will display the username and IP address of the head node for this cluster as depicted in Fig. 4.
Using these credentials, users can login into the remote cluster directly. In *nix OS environments users can run the following command at a shell prompt:
Here <path-to-your-private-SSH-key-file> would be replaced with the corresponding UNIX filesystem path (e.g. /home/rescale/Desktop/id-rsa). If the SSH private key is located at $HOME/.ssh, then ssh will automatically look for private keys stored at that location and the -i flag with its path argument aren’t required. If users are interested in forwarding their remote X-Window display to their local display, then they can invoke the X11 forwarding feature built-in to ssh by passing the -Y flag to the above ssh command. This enables some basic interaction with GUI applications running on the remote cluster for customers if needed.
PuTTY is again utilized to demonstrate establishing the remote SSH connection for users in Windows OS environments. After launching putty.exe, users can specify the location of the private SSH key file stored on their local filesystem by navigating to the Connection -> SSH -> Auth pane and clicking on the “Browse…” button. Next return to the “Session” pane, ensure that the radio button next to SSH in the “Connection type” section is selected, enter the username and IP address provided via Rescale’s job status log message window into PuTTY’s field labeled “Host Name (or IP address)” (e.g. zach_Gcaaa@184.108.40.206), and finally click on the button labeled “Open”. A remote connection session window should open and login to the remote cluster using the credentials provided.
Once your simulation has completed running, be sure to transfer any runtime files back to your local environment, and be sure to click the “Terminate” button on the “Status” page for that particular job to avoid incurring any additional charges.