5 Winning Strategies to Accelerate Engineering – Introduction
Editors note: This post is the introduction to a 6 part blog series. Read the full report here.
Empowering new product development teams to innovate faster is front and center in an enterprise’s strategy to stay competitive. Yet commercializing science & engineering breakthroughs is increasingly difficult. According to Gartner’s 2021 R&D Leadership Council Report, 90% of R&D leaders find accelerating new technologies to support new product development to be difficult, despite it being a top priority.
A primary reason for this challenge is the increasing complexity and sophistication of engineered products – from aircrafts to molecules. The computational science and engineering methods that make these new products possible rely on high performance computing (HPC).
In the broader enterprise, digital transformation has changed the way people work and collaborate, made possible by cloud. However, this transformation is only starting to take place in engineering and R&D, and organizations adopting cloud HPC via a lift-and-shift of existing practices are not realizing the productivity gains they were looking for.
Digitally transforming engineering and R&D requires a fundamental reconceptualization of HPC practices – from focusing on optimization of HPC hardware, to optimization of R&D throughput.
In this new approach, software, hardware, and cloud considerations are all in service to making engineers successful, taking full advantage of capabilities that only cloud can make possible. In this paper, we describe five strategies that represent a fundamental departure from traditional HPC practices that have been proven to help organizations accelerate engineering innovation.
“There is no business strategy without a cloud strategy, and there is no real cloud strategy without paying close attention to the business outcomes that you’re trying to accomplish.”
David Smith Gartner Distinguished VP, Enterprise Architecture & Technology Innovation
Five Winning Strategies to Accelerate Engineering Innovation
From Hardware-centric to User-centric – solving for ease of use for researchers and optimize their productivity.
From Inflexible to Unlimited – providing the broadest optionality on the tools and deployment models that can be used.
From Siloed to Connected – unifying the islands of analysis to enable multi-team collaboration and sharing of best practices.
From Static to Intelligent – making prudent cost-performance tradeoff decisions among the many options available in the cloud.
From Manual to Automated – operationalizing policy-based control to ensure security and compliance while empowering engineers.
State of Cloud HPC
Industry analysts indicate demand for HPC is increasing, with the adoption of cloud HPC accelerating. The global HPC market is estimated to reach $60 billion by 2025 from $40 billion in 2021, according to market research firm, Intersect360. The firm projects that cloud HPC will continue to grow at over a 20% Compound Annual Growth Rate (CAGR) through 2027 to reach an estimated $9 billion. Key factors accelerating cloud HPC include the use of deep learning and AI capabilities, and the need for more flexible architecture options to best run an increasing set of workloads. A key advantage of cloud is the ecosystem that provides faster access to the latest technologies, hardware, and applications.
Data Center Challenges Impact New Product Development
…While An Explosion of New Architectures Are Available In The Cloud
The cloud hit a tipping point for HPC in 2019, according to Hyperion Research. So it’s no longer a question of whether HPC will transition to cloud, but how fast the HPC transition to cloud will be. For the 2022-2023 period, the firm predicted a second tipping point of HPC cloud spending driven by performance, ease of use, and a significant uptick in AI workloads. Organizations are already showing signals of accelerating cloud adoption. Rescale’s 2022 State of Computational Engineering Report found that most organizations (78%) are already using cloud to some degree, while half of organizations are using cloud consistently.
Cloud HPC Spend is Projected to Grow Aggressively
Most Have Started Their Cloud Journey
For organizations that perform engineering or scientific computing
78%
use cloud in some way
53%
use cloud consistently
CIO Priorities for Technology Investments in 2022
The HPC cloud transition to support engineering and R&D is consistent with broader CIO investment priorities. The “cloud-first” approach is no longer isolated to ERP and CRM, and now also encompasses engineering and R&D workloads. Gartner’s 2022 CIO and Technology Executive Survey found that data intelligence, cloud platforms, and application modernization are top investment priorities. Many are going beyond not only using cloud, but optimizing their use of cloud. This requires key capabilities around performance and cost analytics, improving controls, and automating governance, security, and operations.
Technology Investment Plans
Top Cloud Initiatives
Why Engineering Is Accelerating Cloud Adoption
With organizations across the business pursuing digital and cloud transformation strategies, engineering and R&D are also motivated to shift to the cloud. This transition is accelerating as engineering and R&D are finding that their ability to innovate and create new products is severely impacted by traditional HPC practices. With HPC as the engine that powers engineering and R&D, the demand for HPC resources continues to rise. To serve as the business’s growth and innovation engine, engineering needs to be equipped with new capabilities that address several key issues, from managing supply chain challenges to eliminating R&D silos:
From Hardware-centric to User-centric Supply chain issues continue to impact HPC vendors’ ability to fulfill customer orders. The long delay is likely to last for an extended period. As a result, IT is unable to upgrade on-premises HPC infrastructures to meet increasing demand from engineering or any ad-hoc spike computing needs. This has caused longer wait times for engineers, as well as project delays.
Enterprises’ needs for HPC resources are accelerating. As computational science and engineering methods see broader adoption, HPC is seeing increased demand. Additionally, organizations using simulation are onboarding new workloads and adopting additional techniques such as multiphysics and surrogate ML models, further driving up demand for HPC resources.
Talent shortage for HPC is impacting both HPC vendors and customers. Many enterprises are finding that they do not have sufficient expertise to implement new technologies, slowing down their IT modernization efforts. Vendors meanwhile are not in a much better position to help.
Proliferation of islands of analysis in organizations. Since traditional HPC approaches employ tightly-coupled systems, R&D teams create workload-specific technology stacks focused on their needs, generally disconnected from each other. The result has been slower product innovation and less collaboration across R&D teams.
Supporting increasingly distributed workforces. Organizations have discovered that remote work is often feasible in a post-pandemic world. For engineering and R&D, the need is to provide anywhere access and better collaboration capabilities that provide flexibility and attract more talent.
“The demand for technology talent maintains a frenetic pace with an emphasis on data and analytics, information security, architecture, cloud and engineering.”
Craig Stephenson, Managing Director, North America Technology Officers Practice at Korn Ferry
Traditional HPC Limits Innovation in Cloud and On-Prem Environments
Businesses recognize the need to bring new product innovations to market faster, and that doing so enables capturing greater market share, and reduces the risk of being disrupted by competitors. Gartner’s 2021 R&D Leadership Council found that for R&D leaders, accelerating new technologies for new product development is a top priority. However, 90% of the same R&D leaders found doing this difficult.
There are several reasons for this. Bringing engineered products to market makes use of computational science & engineering methods, powered by high performance computing (HPC). And HPC is a discipline that has not seen major changes in decades. Since HPC first emerged from supercomputing, it has been defined by managing expensive and complex specialized architectures.
This mindset of maximizing utilization, as opposed to maximizing R&D throughput, remains a defining characteristic for many HPC organizations – including some who are migrating HPC workloads to the cloud in a lift-and-shift approach.
90%
of R&D leaders find accelerating technologies for new product development difficult, despite being a top priority
Source Gartner R&D Leadership Council 2021
Lift-and-shift cloud HPC presents several challenges. First, the fundamental benefits of cloud – near-infinite scale, and a broad range of architectures – gets ignored. An organization doing traditional HPC with lift-and-shift to cloud may no longer need to maintain data centers, but all other aspects of infrastructure maintenance, security and administration will grow in complexity. All of this without benefiting from cloud’s scale and architecture options.
Second, taking a lift-and-shift mindset to cloud ignores another fundamental characteristic of cloud, which is its connectedness. In a world where nearly all resources are accessible via APIs, new opportunities around collaboration, data sharing, and continuous integration become possible. But doing this requires a storage and data strategy that is cloud-aware and user-centric, instead of creating yet more islands of analysis and data isolation.
Lastly, taking advantage of cloud requires unprecedented levels of automation and analytics. Cloud’s scale and dynamic nature means more possibilities, but can also require new skills that many organizations may not have. Traditional approaches using scripts and one-off configurations become untenable very quickly as new teams are onboarded and new applications are implemented.
The only way for organizations to turn HPC from being a bottleneck into a force multiplier for new product innovation, is to employ new levels of automation to harness the full potential of cloud operating models to support R&D.
HPC Built for the Cloud Enables Engineering and R&D Transformation
Business technology leaders are looking to adopt cloud HPC to bring digital transformation to engineering and R&D. Having seen the power of cloud to transform other business functions such as CRM and software development, many have now set their sights on driving the same process and productivity improvements for engineering and R&D. There are lessons we can take from other cloud recent enterprise digital transformations, that inform how R&D transformation will likely need to occur. Although cloud was an enabler of these transformations, the key productivity improvements came from changing how users and organizations interacted – not from where the compute was running.
Lessons from successful digital transformations tell us that it is a fallacy to believe that doing a lift-and-shift of existing HPC practices to cloud will bring measurable productivity improvements in R&D.
For example, in software development, new practices of open source and social coding tools like Git and Jenkins CI/CD, enabled by cloud services, accelerated development. In CRM, cloud-enabled API interfaces created cloud-based SaaS ecosystems, which in turn brought unprecedented levels of automation and analytics to improve how sales and marketing work together.
The engineering and R&D transformation that technology leaders seek will therefore not come from cloud, but from what cloud enables. In Rescale’s 2022 State of Computational Engineering Report, the study found that giving engineers access to infinite compute is helpful, but what really drives productivity is making access to compute easy.
Technology leaders therefore need to bring SaaS-like simplicity against a traditionally complex HPC stack, while harnessing the scale and variety of cloud architectures. This means starting from the R&D operating model they want, and finding how the cloud can deliver it.
Impact of computational tools on project success probability
HPC Built for the Cloud is Distinct from Traditional HPC in Five Fundamental Ways
Traditional HPC approaches can be used in data centers or cloud, but the net effect is the same – in both cases, it starts with making infrastructure decisions, with downstream implications for the applications used and how users work and collaborate.
HPC built for the cloud, on the other hand, starts with the user. All the resources that the user needs are instantiated and accessed on demand. These resources can range from hardware (from any cloud provider), to application (from any software vendor), to data (from anywhere within the organization), to best practices (shared by other users). User objectives dictate the application that is used. And the applications used dictate the hardware architecture used, based on cost vs. time-to-solve objectives.
A built-for-the-cloud HPC approach is user-centric, unlimited, connected, intelligent, and automated. We will explore each in the following sections.
Editors note: This post is the introduction to a 6 part blog series. Read the full report here.
