Giving R&D Teams a Home in the Cloud for Their Homegrown Engineering Software

Today, research and development teams rely on a variety of tools to simulate and analyze complex physical, chemical, and biological interactions that happen in the real world. 

But to do that, R&D organizations need two things: fast computers and sophisticated software. Thanks to the cloud, open-source software, and other trends, computational researchers and engineers today have far more choices for both their computing power and applications.

While these developments are great news, this expanding set of options is creating new management challenges for organizations that want to accelerate their pace of innovation.

Rapidly Expanding Choices in R&D Hardware and Software

The continuous research and development of compute-intensive simulation, modeling, and other computer-aided engineering (CAE) tools pushes the boundaries of hardware performance. But those boundaries are rapidly expanding, thanks to the development of cloud computing and new, specialized computer chips. With the cloud, most any R&D team can now quickly plug into the high performance computing (HPC) cluster of their choice to drive their computational research efforts.

While the democratization of ultra-fast supercomputing clusters is revolutionizing R&D, there’s a simultaneous explosion of software taking place that’s changing the composition of science and engineering toolkits. Specifically, product-driven and R&D-focused organizations are using more homegrown and customized third-party applications for computational research and engineering. 

A few key industry trends are propelling a rapid proliferation of software portfolios: collaboration across supply chains, custom AI/ML-enhanced workflows, and open-source tools and data sets. Especially as engineering and design problems become more multidisciplinary, computing workflows are being tailored to company-specific problems (e.g. simulating physical, chemical, and electromagnetic characteristics in the same study).

Using custom engineering software is not a new concept, but now the accessibility and practicality of cloud-based HPC give engineers and scientists new possibilities to test, scale, and share a larger number of R&D tools than previously possible.
 
Cloud HPC delivers flexibility in both hardware variety and capacity, which allows practitioners to accelerate their tools and perform their simulation, modeling, and analysis from anywhere. As a wave of new specialized architectures becomes available, each workload can be configured on the architecture that is best suited to its requirements.

Why Custom Software Is on the Rise

As companies develop expertise and unique approaches to R&D, they often create custom software for their proprietary data and research methods. The use of custom tools, from open-source to in-house developed packages, is becoming increasingly commonplace, especially as organizations invest more in compute-driven R&D. 

While commercial (licensed) applications make up a steady share of computing consumption (and budgets), Rescale has witnessed customer preferences shift toward more variety and customization in their R&D tools, for example: solvers, compilers, simulation and modeling tools, frameworks, interactive workstations, visualization tools, post-processing tools, workflow automation scripts, SDKs, libraries, and open-source tools. 

The use of homegrown and customized open-source HPC software is emerging as a competitive advantage for three key reasons:

• Custom software can harness an organization’s intellectual property and unique data sets to better test and refine their products or research methods. For example, custom AI models trained on existing customer data or unique product behaviors can support physics-informed neural networks and generative design technologies. When combined with traditional CAE, these techniques can produce better products faster. 
• Custom software can be more cost-effective than commercial software. It does not require licensing for each user, opening up access to virtually any number of users. Historically, commercial software made up the majority of engineering and scientific research workloads, with licensing often exceeding all other line items in the computing stack. Ongoing cost increases have caused many organizations to look for ways to reduce their licensing costs by leveraging free tools. New open-source software and homegrown tools are quickly becoming competitive alternatives. 
• Custom software can take advantage of new hardware configurations to significantly accelerate performance and efficiency. As software options proliferate, hardware options and compatibility have also greatly increased. Open-source toolkits and frameworks such as NVIDIA’s CUDA, Google’s Tensorflow, and AMD’s ROCm can now be used to accelerate HPC applications on their specific computing architectures. For example, Rescale customers have reduced the time required for digital simulations by 200% while reducing computing costs by 80% – all while using a free-to-use CFD solver from NASA called FUN3D. 

Common Challenges Managing Software Sprawl

To keep up with software innovation, IT managers need to keep pace with the growing complexity of supporting and maintaining larger software portfolios. In an era when everyone is now accustomed to instantly accessing cloud-based software, providing a similar experience with homegrown software is a challenge. 

It’s true that cloud-based HPC has great potential to help scientists and engineers quickly scale their custom applications, but the path to getting there can be challenging for teams unfamiliar with both application and cloud management. Each software and workflow has unique requirements for configuration and access, which can take time and repeated manual steps, likely leading to costly delays in R&D efforts. 

Some organizations allow their R&D teams to onboard new software independently. But even if these employees have the skills to do so, unmanaged and decentralized software publishing can pose significant risks to IT governance, such as cost overruns, security vulnerabilities, or violating industry compliance standards. 

One potential solution to self-service HPC software deployment is through containerization, which can get the job done with some drawbacks and risks. For example, containers require familiarity with a variety of new tools, pose root access security vulnerabilities, and usually diminish software performance. Consequently, R&D in most organizations is entirely dependent on IT to deploy new software into production. 

Accelerating Software Deployment and Unifying Governance

Rescale is committed to supporting a growing catalog of commercial and open-source software. As part of that ongoing effort, we are excited to announce Rescale Software Publisher. Software Publisher addresses the opportunity and challenges organizations now face in publishing homegrown R&D software to the cloud. 

Publisher gives authorized users and admins the ability to “self-publish” to their private Rescale software library — it’s fast, simple, and creates a robust image that can be scaled consistently across the business and on your cloud of choice. Engineering teams can now leverage their unique set of R&D tools to commercialize new products faster.  

Key features of Rescale Software Publisher include:

• IT and authorized R&D users can easily deploy any custom, open-source, or commercial software available to team members, consistent with IT policy.
• Automated configuration ensures consistency, portability, and performance across the best, latest architectures.
• Centralized software portfolio management unifies visibility and control of cost, access, security, and compliance.

Companies running any custom compute-intensive scientific or engineering R&D software can start taking full advantage of the cloud, from testing to full-scale deployment, in just a few quick steps. Whether you’re an R&D or IT user, Rescale Software Publisher’s interactive sandbox gives you a simple interface to validate your software package and onboard it into your private software library like any other Rescale-supported package.