Feature Release: Job-Oriented File Management for Modern Cloud HPC

Accessing and managing HPC job input and output files in the cloud is foundational to R&D productivity. As part of ongoing platform improvements, we are excited to share new capabilities within Rescale Files, which enhance how users manage their simulation and modeling data. This release features a new job-oriented capability for accessing, organizing, and sharing job-specific files in-context, enriched with a variety of intuitive capabilities and metadata fields.

Rescale Files is a core component of our HPC as a Service foundation, with intuitive data management tools that accelerate HPC and AI workflows, facilitate seamless collaboration, and maintain job integrity across complex projects.

Rethinking File Management for Accelerated Workflows and Collaboration

For engineers, scientists, and researchers who use high performance computing (HPC) to drive their analysis, managing and sharing files across multiple jobs can be a complex and time-consuming task. Modern HPC workflows often consist of multiple jobs with file dependencies from previous jobs. This makes integrated cloud file management a critical necessity to maintain productivity, collaboration, and security. 

Traditional approaches to file management often break down as HPC jobs accumulate large quantities of files, often with files sharing the same name. This results in users struggling to locate specific job data or not knowing which files should be deleted or kept.

Workflow of computational pipeline with job output result files being used as inputs for subsequent analysis

Your Files In Context and New Capabilities

With additional job-oriented context and new interface capabilities, users can now easily find files through searching, filtering, and sorting with various criteria such as job name, file name, file type, and date range. 

Rescale Files also allows users to delete files individually or by entire directory to manage storage consumption, with the option to recover deleted files from the trash during a user-defined retention period. Users have full control to set a job retention period to automatically delete jobs and their associated files after the retention period has expired.

Rescale Files ‘My Jobs’ view with job output files associated from ‘Airfoil Test 2’

Ensuring HPC Job Integrity and Efficiency in Data Management

One of the key benefits of Rescale Files is in helping teams preserve job file dependencies to ensure job integrity and prevent data loss. Users can use completed jobs as the starting point to find relevant result files, either to be used as input for new jobs or to download them for further processing. Within a shared Rescale Workspace, collaborators can clone or share jobs, making it easy to compare results and automate multi-job workflows. 

Benefits of Rescale Files include:

  • Organization: Users and administrators can store and find files where they need them, in context with the jobs they are associated with.
  • Job integrity: Job file dependencies are preserved, and critical files are protected.
  • Collaboration: Users and their jobs have access to the right files.
  • Storage efficiency: Organizations can safely and systematically delete files to control storage costs.

Start Using Rescale Files Today

Customers can request an upgrade to the new version of Rescale Files at no additional charge. Once upgraded, users can follow in-platform tool tips to begin exploring new features. These new features and automatic upgrades for existing customers will go into effect later this summer. 

Contact us today to take advantage of Rescale’s streamlined, flexible file management to help improve your HPC workflows.


  • Garrett VanLee

    Garrett VanLee leads Product Marketing at Rescale where he works closely with customers on the cutting edge of innovation across industries. He enjoys sharing customer success stories, research breakthrouths, and best-practices from Rescale engineers, scientists, and IT professionals to help other organizations. Garrett is currently focused on the convergence of supercomputing, HPC, and AI simulation models and how these trends are driving discoveries in science and industry.

Similar Posts