Rescale: Minimizing Data Challenges

When performing HPC in the cloud, one of the workflow tasks engineers and scientists have to deal with is the handling of large amounts of data. The question the engineer often has to ask is: “What data is useful to me and how can I present it in a functional form?” When all the data is stored in the cloud, it is essential to minimize the time between generating the data and being able to practically use the data. Simply downloading gigabytes of data is not the preferred or most efficient way.
Of course, different applications strive towards different goals. Sometimes the engineer wants all the output data stored locally and readily accessible. Other times, the cloud proves to be a convenient way to store data for projects and the engineer can, at any time, download date a subsets of jobs run months ago; however, turnaround time for an HPC job is essential. There are several approaches to minimize the turnaround time for data-intensive jobs. Below are some real ways Rescale customers minimize the time by optimizing how they handle data.
LS-DYNA pre/post
LS-DYNA jobs can generate large data output. It’s up to the user what data their job generates. How often the run writes to the hard drive and what data is written is left to the user to configure. However, the average engineer is not 100% sure what data he’ll need, but thinks he has a good idea of what he’ll end up using. The engineer makes the decision to save as much data as possible  to binary files and knows how to use Rescale’s convenient wrapper script to run his LS-DYNA jobs:

ls-dyna -n  -s ; -i  -p

The engineer, however, saves himself download bandwidth by using Rescale’s built-in command-line post-process option and using the ‘-d’ flag to post-process his input files into a single .db file:

ls-dyna -n 16 -s 4 -i big-model.k -p single -d create_postdb.inp

Because he knows the information he will likely need (displacement, plastic strain, Von Mises stresses and thickness), his create_postdb.inp file may look something like this:

d3plot
 output.db
 state_on = 10, 20, 30, 40, 50, 60, 70, 80, 90, 100
 displacement
 plastic_strain
 von_mises_stress
 thickness

The engineer now only has to download output.db with all his desired information in it. So, instead of 35GB of data he is downloading a single 1GB file. Instead of waiting an hour and a half to download his file, the time is reduced to just a couple minutes to download all the data he needs.
On-cluster post-processing
Another engineer is running Converge CFD. Her job just completed and has generated 40 GB of data. Fortunately, she has set up her ssh credentials on the Rescale settings page:

She’s entered her public key–which Rescale has put on all her Rescale nodes–and set up a CIDR Rule so her nodes are only accessible from her network. The job logs have notified her how to connect to the cluster:

She can now easily ssh into her cluster by copying and pasting the command in her terminal or using putty on Windows. When logged into her node, she simply runs:

post_convert

This launches Converge CFD’s convenient post-process tool which allows her to convert select sets of data into Tecplot, Ensight or other plot-able formats. She can now use scp or sftp to download these small plot files to her local hard-drive.
I don’t want to ‘sync’ about downloading all my data
The first engineer runs many jobs on Rescale, but when he comes in to the office each morning, he realizes that many of his jobs completed overnight. Fortunately, instead of having to go and manually download all his output files, he ran Rescale’s convenient java-based command-line utility for downloading files to his hard-drive whenever a job finishes.
After requesting an API key from Rescale, he launched the command-line utility from his work computer using the simple command:

java -jar rescale.jar sync -p abcdef1234567890 -d 600

The command uses Rescale’s API to check every 10 minutes whether there are any newly completed jobs; and if there are, Rescale will download the files automatically to a local drive. The command-line utility efficiently downloads and decrypts (multi-threaded) the output data. So, while our engineer was sleeping, his company network was hard at work downloading all finished jobs to his work computer so he could spend the next day being more productive.
Coming soon: GUI based post-processing
Although the second engineer can already use X11 forwarding over ssh to do some basic GUI based post-processing in the cloud, she thinks it would be very useful if the performance was better. Fortunately, she won’t have to wait long as Rescale is actively working on a remote desktop solution for doing GUI based post-processing
Being smart with data
The bottom line, when it comes to data in the cloud, is to use the tools at your disposal as efficiently as possible. LS-DYNA and Converge CFD are not the only software providing easy to use command-line post-processing tools. Many other tools available on Rescale provide this same functionality.
The question of “What is done in the cloud and what is done locally?” is only answered by deciding how Rescale fits best into your existing engineering processes. Sometimes it is convenient to have all data on premise while doing all post-processing on-premise as well. But often, more of this engineering work can be moved to the cloud as demonstrated in both engineering cases.
For this reason, our goal at Rescale is to provide all the necessary tools to make the issue of dealing with data as minimal as possible.