Feature Release: Job-Oriented File Management for Modern Cloud HPC

Accessing and managing HPC job input and output files in the cloud is foundational to R&D productivity. As part of ongoing platform improvements, we are excited to share new capabilities within Rescale Files, which enhance how users manage their simulation and modeling data. This release features a new job-oriented capability for accessing, organizing, and sharing job-specific files in-context, enriched with a variety of intuitive capabilities and metadata fields.

Rescale Files is a core component of our HPC as a Service foundation, with intuitive data management tools that accelerate HPC and AI workflows, facilitate seamless collaboration, and maintain job integrity across complex projects.

Rethinking File Management for Accelerated Workflows and Collaboration

For engineers, scientists, and researchers who use high performance computing (HPC) to drive their analysis, managing and sharing files across multiple jobs can be a complex and time-consuming task. Modern HPC workflows often consist of multiple jobs with file dependencies from previous jobs. This makes integrated cloud file management a critical necessity to maintain productivity, collaboration, and security. 

Traditional approaches to file management often break down as HPC jobs accumulate large quantities of files, often with files sharing the same name. This results in users struggling to locate specific job data or not knowing which files should be deleted or kept.

Your Files In Context and New Capabilities

With additional job-oriented context and new interface capabilities, users can now easily find files through searching, filtering, and sorting with various criteria such as job name, file name, file type, and date range. 

Rescale Files also allows users to delete files individually or by entire directory to manage storage consumption, with the option to recover deleted files from the trash during a user-defined retention period. Users have full control to set a job retention period to automatically delete jobs and their associated files after the retention period has expired.

Ensuring HPC Job Integrity and Efficiency in Data Management

One of the key benefits of Rescale Files is in helping teams preserve job file dependencies to ensure job integrity and prevent data loss. Users can use completed jobs as the starting point to find relevant result files, either to be used as input for new jobs or to download them for further processing. Within a shared Rescale Workspace, collaborators can clone or share jobs, making it easy to compare results and automate multi-job workflows. 

Benefits of Rescale Files include: