Transferring Job Files to and from Rescale

Rescale platform offers several ways to transfer files to your local workstation

After your job has finished, the Results pane displays all the files associated with the job.

Go to the Results pane for your job

results-pane-main.png

On the Results pane, the options to download files are:


Download entire job


Click on Download Job to download a zip file containing all files associated with the run, including input and output files

results-pane-download-job.png

Search for files


Type the file name or extension in the search bar. Click on Search. Click in the check box. Click on the Actions dropdown button and then click Download selected files

Note: If a single file is selected, the file itself will be downloaded. If more than one file is selected, the files will be compressed first and then download (the compression may take a few minutes before the download begins).

Note: You can search by filename or by the extension. Wildcards are currently not supported. For instance, to search for process_output.log , you can type either process or .log

results-pane-search-file.png

By file(s) selection


Manually select the desired file(s) by clicking in the check box. Click on the Actions dropdown button and then click Download selected files

results-pane-manual-select.png

By direct download


Download individual files by clicking on the Download icon

results-pane-download-icon.png

The Files tab on the menu bar shows all files from all the jobs previously run from your account.

Go to the Files tab on the menu bar

files-tab-main.png

The options are:


Search for and download files


Type the file name or extension in the search bar and press Enter. Then, check the box next to any file and select the Download selected files as a zipped file icon. Compression may take a few minutes before the download begins.

Note: You can search by filename or by the extension. Wildcards are currently not supported. For instance, to search for all files surface_flows.csv, you can type either surface_flows or .csv

Search Filters

In addition to entering search criteria by name or extension, you can also filter your search results with the following options:

  • Date: From the date picker, select a date or date range of when the file was created.
  • Type:  Select from the following file types: Any type (default), Input, Template, Param, Script, Output, Case, or Checkpoint.
  • Files: Select files by owner: Any owner, My files (default), Shared with me
  • Extension: Select the + icon and enter an extension on which to search. Any extension is selected by default.

Manually select and download files


You can also manually select files by checking the box next to any file in the list and selecting the Download selected files as a zipped file icon.

The Snapshot feature allows you to take periodic snapshots of the working directory for a run of your Rescale job, either for restart purposes or for additional review on your local machine.

When your job is running, the Status Tab will have a Live Tailing window. If you are running a job with multiple runs, such as a DOE, there may be multiple runs in the Runs column.

  • Select the run of your choice from the Runs column
  • Click on the Snapshot button to create a snapshot of the current working directory, including subdirectories
snapshot-status-page.png
  • All files are packaged into a single zip file. Once that zip file is ready, a download button will appear, as shown below. Click on Download
snapshot-popup.png

Snapshots that you create are also available for later download from the Results tab. If you created multiple snapshots over the course of the run, each one will be listed and labeled with the date and time it was taken.

snapshot-results.png

The “scp” linux command allows you to securely copy files from the compute cluster to the local workstation while a job is currently running.

The steps are:

  • Set your SSH Public Key
  • Determine the files you want to copy and the destination on the local workstation. The job files are located in the ~/work/ directory. You can also connect to your cluster using Rescale’s In-Browser SSH or Manual SSH to verify the location of the file.
  • Use the following command line on your local terminal to transfer files :
scp -i ~/.ssh/<private key> <source> <destination>
  • To access the cluster directory, we must specify the cluster ID, which can be found as shown:
scp-ssh-cluster.png
  • Here is an example for transferring files from the cluster to your local workstation
Private keyid_rsa
Location of SSH key~/.ssh/
Cluster IDxxxx_xXxxxx@99.99.99.99
Source Directory~/work/
File to copyprocess_output.log
Destination/Users/johndoe/cfdruns/
  • In the terminal type, type the following for the above example
scp -i ~/.ssh/id_rsa xxxx_xXxxxx@99.99.99.99:~/work/process_output.log /Users/johndoe/cfdruns/
  • The file process_output.log is transferred from the cluster to the folder cfdruns.
  • You can transfer multiple files with similar name using the asterisk symbol *. For example, to transfer files – velocity-contour-0, velocity-contour-1, velocity-contour-2, velocity-contour-3, velocity-contour-4, the scp command will look like –
scp -i ~/.ssh/id_rsa xxxx_xXxxxx@99.99.99.99:~/work/velocity-contour-* /Users/johndoe/cfdruns/
  • If you are copying a file from your local workstation to the cluster, the scp command will look like this –
scp -i ~/.ssh/id_rsa /Users/johndoe/cfdruns/FileToTransfer.txt xxxx_xXxxxx@99.99.99.99:~/work/
  • The above command will transfer the file FileToTransfer.txt from the local workstation to the work directory of the cluster.

WinSCP is a popular file transfer client for Microsoft Windows. You can download WinSCP from here. Please follow these steps to transfer using WinSCP –

  • Create your public/private key pair and copy them to a folder you can access in your local workstation. For information on creating SSH keys, please refer to Manual SSH > Windows user > Generating your SSH keys in Connecting to your cluster resource page.
WinSCP Keys
  • Open WinSCP and fill out the login information from Rescale as shown below. Select File protocol as SFTPHost name with the IP XX:XXX:XX:XX and User name with the job ID uprod_xxXxX. The IP and job ID can be found on the Rescale platform as shown below
WinSCP Login
  • Click on Advanced > Advanced… and then click on Authentication in the tree list on the left. Under the private key file field, browse for the newly created private key.
WinSCP Advanced
  • Accept the changes and click on Login. Your public key should now be authenticated and you will be connected to the cluster as shown below.
WinSCP Authernticating
  • You can transfer files between your local workstation (on the left) and the cluster (on the right).
WinSCP Window

The Rescale Transfer Manager is a native Windows application that can be used to download output files from jobs. This is a more robust and faster alternative to downloading large files through your browser.

At the moment, Rescale Transfer Manager is only available on Windows Operating Systems.

For more information, please refer to the page here.