What Wheat Farming Can Teach Us About High Performance Computing

Today we are excited to announce a set of features that collectively represent a key milestone for cloud operations, and enable a new level of holistic automation only available on Rescale.  We call it Intelligent Resource Automation – read the announcement. I’d like to share why this milestone is a major step for the digital transformation of engineering and R&D.

From Farm Acres to Core Hours

Not all organizations undergo digital transformation at the same time or pace, but at some point it will happen in pursuit of growth. In my personal experience growing up doing farm work, even our modest farm needed digitization to remain competitive. Our operations coverining 3,000 acres of wheat involved the cultivation, processing, and distributing for our own and many other farms’ crops. During our own digital transformation, I witnessed firsthand the impact of instant access to data like inventory counts, tractor field paths, and delivery timelines. Overall, our digital transformation increased our product yield, customers grew happier, and the business expanded.

While farm life is hardly high performance computing, both are operations-intensive businesses where optimizing resources plays a key role in the business’s success. From clipboards and pencils to a shared office computer and eventually cloud ERP and digitally connected harvesters, we eventually reached a level where we could make data-driven decisions about our entire operation. An acre of farmable land is analogous to a core-hour of capacity on any cloud provider anywhere in the world. That core hour might be best suited for specific workloads, may exist in specific geographies, each at different price points. 

Today, many organizations are undergoing a similar transformation in their cloud HPC practices as they look for ways to access precious computing resources (e.g., a high-demand GPU), and unify disparate systems, teams, and data sources to optimize operations. Automation made it possible to scale our farm and similarly automation is enabling HPC operations to scale as they take on more projects and support global needs.

Growing Cloud Operations Draw Increased Scrutiny

Technology and engineering leaders are under increased pressure to report on and forecast the current and future state of cloud resources and their efficiency. Simultaneously, systems administrators and security & compliance managers are anxiously warding off breaches and upholding industry and regional compliance standards. Despite having access to virtually limitless computing potential in the cloud, our customers told us that they were still running into manual processes, disconnected islands of analysis, and lack of operational visibility.

The benefits of cloud computing being distributed include that it is democratizing and able to add new users and provision isolated sandboxes. Organizations with cloud sprawl can be overwhelmed after a lift-and-shift approach when they are surprised at the complexity of cloud HPC operations. IT departments and HPC practices need ways to manage the many computing resources available (HW capacity, SW licensing, users) and maintain a range of policies (security and compliance, budgets, access). The challenge is that these capabilities are not possible with traditional HPC approaches, and they are challenged to connect to the broad ecosystem of tools that comprise many modern HPC stacks.

Solving HPC Complexity With Intelligence and Automation

We set out to solve the growing problem of complexity and resource management in computing operations at any scale. Rescale’s cloud control plane approach enables customers to unify any cloud and any workload with analytics to gain a full-stack understanding of HPC hardware and software, including current consumption, spend, and cost-efficiency.

Born out of Rescale’s co-founders’ need to aggregate computing resources to optimize commercial airliner designs, our technical DNA is geared towards helping customers unify their resources, understand their data, and take action on it. In 2020, we launched business management capabilities for IT and HPC pros in order to give them visibility and control over their computing operations. With this release, Rescale advances those capabilities with automation based on computing resource intelligence to ensure smooth business continuity while minimizing risk to the business. 

Global and Enterprise-Grade HPC Management for Any Scale Organization

Intelligent Resource Automation capabilities on Rescale empower science and engineering teams that depend on HPC while ensuring IT control to limit business risks. Administrators can now set policies together with engineering, security, and finance stakeholders to ensure best practices are followed and compliance standards are met. With the ICA feature set, Rescale account administrators can harness:

• Software license-aware queuing – Maximize license utilization, increase engineering productivity, and reduce overall computation cost and wait times. 
• Multi-architecture resource prioritization – Give engineers the best-performing architectures while ensuring business continuity regardless of capacity constraints or regional outages of specific architectures.
• Multi-service level priority – Deliver the best cost-efficiency and quality of service to meet deadlines or budgetary goals by automating workloads between Rescale’s enhanced infrastructure service levels.
• Global geo/region priority – Ensure data locality requirements are met while minimizing wait times and execution failure caused by regional capacity constraints. 

These capabilities collectively are truly unique to Rescale and designed to help all of our customers capture more value from their investments in computing and talent. We believe that intelligence and automation are key to accelerating timelines and driving efficiencies. Similar to how farmers say “make hay while the sun shines”, HPC engineers have tight deadlines to achieve new R&D milestones. Thanks for bearing with my personal anecdotes – it’s not often that I get to draw parallels between my early career in the wheat fields and current one in technology!

You can learn more about these features on the Intelligent Resource Automation overview page and documentation pages. We look forward to hearing from you if you have questions or are ready to get started.