This page shows you how to manage all the generated output files from your jobs. Effective management of your output files has numerous benefits, including reduced file size (and hence transfer speed), effective housekeeping of data, and easy access to files on Rescale Cloud Files.
When running a job on Rescale, the working directory is located at
$HOME/work/. This is where the running software dumps all its output files, as well as the working directory of the command line prompt that you are exposed to by default.
Once the job has run to completion, the Rescale platform saves and transfers everything that is in the aforementioned work directory. This is usually the last step in a job, and the time taken to do so is dependent on the total file size and number of files.
Removing unwanted files
A straightforward way to effectively manage your files would be to remove unwanted files at the end of your job. You can do this by appending additional commands below your simulation command.
A quick tip would be to use the wildcard
* to match any character and delete multiple files in one command.
A sample command to include in your software settings page is shown below:
Please note that deleted files cannot be recovered.
Compressing your files
There are a number of benefits to compressing your files. Firstly, it lowers the file size and number of files, thereby speeding up the upload and download process. This is particularly important when your output files are expected to be very large (>10GB). Secondly, compressing your work directory into a single compressed folder helps preserve your directory structure. If you are managing your files on Rescale Cloud Files, it will be advantageous to store zip files instead if you want to keep the directory structure.
Rescale supports the following compression types:
- .tar, .tar.gz, .tgz, .gz
For example, if you want to archive all your d3plot files, you can use a command similar to:
Output File Filters
This feature makes downloading a subset of your job result files easier by separately archiving files that match a filter. Filters on Rescale utilize regular expression (regex) to match and return an output.
Setting up Selector(s)
You can access the Filters button from anywhere throughout the Job Setup as long as your job has not yet been submitted. Click on the Filters button, then an Output File Archive Filters window will appear as shown in the figure below:
Write your filter in the Selector box and click Add to apply the selector. The applied filter will be shown in the selector(s) list. You can add as many file filters as you need. For multiple filters, it will take all matches and place them into a single archive.
If you need to remove the selector, simply click the X key under the Action column. Below are several examples that can be useful for your case.
Example file filters everywhere for all directories:
This selector will zip all files in the results directories
This selector will zip all files that are named exactly
velocity.jpeg everywhere in the results directories
This selector will zip all files that end with a
.out extension everywhere in the results directories
This selector will zip all files that begin with
residual everywhere in the results directories
This selector will zip all files that contain the string
plot in the file name everywhere in the results directories
Example file filters for a specific directory:
This selector will zip all files that end with
.jpeg extension in the Results root (top level) directory
This selector will zip all files inside the Airfoil directory
This selector will zip all files inside the Airfoil directory that end with
Finding zipped Selector(s)
Once your job is completed, go to the Results page. All output file filters will be archived in a zip file and named
output-archive.zip. This file can be found in the root (top level) directory of Results files as shown in the example image below.
You can also utilize post-processing scripts to run more complex cleanup and post-processing commands. This script will only be executed once all your commands in the software settings page have run. You can use this feature/script to clean up your job output, or parse out variables for easy viewing in the results page.
You can read more about using our post-processing feature here.