What is a HPC Data Center? HPC Data Centers 101.

HPC data centers

HPC Data Centers (High-Performance Computing) are environments or facilities designed specifically to host extremely powerful computer entities often called supercomputers. There can be many different use cases for such a data center, ranging from commercial to engineering or scientific and/or research. The biggest common denominator of all these use cases is the need for fast data processing capability and immense processing power.

Most of the large-scale IT and cloud-computing enterprise leaders are using HPC on a regular basis – Emergen Research claims that some of the companies in question are Microsoft, Dell, Intel, IBM, Amazon, and Atos. The following verticals represent  some of the biggest users in HPC:

• Healthcare
• Fintech
• Government
• Defense
• Research laboratories
• Oil and gas industry
• Entertainment

HPC data centers: necessary elements

There are multiple different parameters or characteristics that have to be included in data centers in order for them to be truly considered HPC Data Centers, such as:

• Since HPC data centers utilize specific high-performance storage solutions (parallel file systems, for example), it is rather common for HPC storage to have a requirement of high-speed access to stored data, as well as vast amounts of data storage capacity.
• Data centers have always had a reputation for significant energy consumption, using a lot of energy with their high-intensity, high-performance hardware. As such, there is always the need for various energy efficiency improvement measures, including more specific power management strategies, optimization in terms of cooling systems, implementation of renewable energy sources, efficient use of excess heat, and more.
• HPC data centers serve as the hosting grounds for clusters of high-performance computers, they are equipped with robust CPUs, substantial memory capacities, and dedicated accelerators such as GPUs (Graphics Processing Units) or FPGAs (Field-Programmable Gate Arrays). These computing systems are finely tuned for concurrent processing, allowing them to execute intricate computations and simulations with significantly higher efficiency compared to conventional computer systems.
• High-performance computing systems produce substantial heat as a result of their potent hardware components. In order to sustain ideal operational environments, HPC data centers employ state-of-the-art cooling mechanisms, including liquid cooling or immersion cooling. These kinds of cooling measures are essential in order to effectively disperse heat while keeping energy consumption to a minimum.
• Considering the paramount importance of numerous HPC applications, data centers must guarantee elevated levels of reliability and accessibility. This objective is met by implementing fault-tolerant architecture, redundant power sources, backup generators, and comprehensive disaster recovery procedures.
• HPC data centers are equipped with high-capacity, low-latency network infrastructure to facilitate swift communication between computing nodes and storage systems. This is vital for supporting parallel processing and the efficient management of extensive datasets.
• Due to the critical nature of numerous HPC applications, data centers are compelled to uphold exceptional levels of reliability and accessibility. These goals are accomplished by implementing fault-tolerant design, redundant power sources, auxiliary generators, and comprehensive disaster recovery strategies.

HPC data centers: monitoring and management

The management and oversight of HPC data centers encompass an extensive range of procedures and utilities aimed at ensuring the utmost performance, dependability, and effectiveness of the facility. This encompasses the continuous monitoring of the well-being and functionality of computing hardware, cooling systems, network infrastructure, and storage solutions. Additionally, it involves tracking energy usage and environmental conditions.

Cutting-edge monitoring systems gather and assess real-time data on diverse parameters like temperature, power consumption, and network activity to detect potential issues and enable proactive maintenance. Furthermore, administrators leverage automation and orchestration tools to streamline the setup, arrangement, and administration of computing resources, software, and workloads.

These tools play a vital role in preserving elevated levels of system accessibility, optimizing the use of resources, and reducing downtime, all while ensuring that the HPC data center operates within the predefined performance and efficiency criteria.

Here are a few examples of applications and use cases that require HPC data centers:

• High-performance computing expedites the drug discovery and development process through the simulation of molecular interactions. This aids researchers in gaining insights into diseases at the molecular level, ultimately leading to the creation of more potent treatments and therapies.
• HPC data centers deliver the substantial computational capabilities required for training advanced machine learning models, catalyzing advancements in AI research, and facilitating the creation of state-of-the-art algorithms spanning multiple industries and applications.
• HPC data centers also play a pivotal role in simulating intricate climate systems and weather patterns. This contributes valuable perspectives on climate change and extreme weather phenomena, thus guiding decisions related to environmental management and sustainability policies.

HPC data centers are quite expensive on their own, and the data they process may be even more valuable. As such, a robust security system is also a necessity – especially when it comes to operations such as backup and recovery (since HPC data centers deal with large amounts of data that is transferred with an incredible speed). One good example of a backup and recovery solution that is capable of working with HPC data centers is Bacula Enterprise.

Bacula Enterprise and HPC data center backup tasks

Many organizations currently utilizing HPC are facing an increasingly complex IT landscape. This complexity arises from the constant movement of data between on-premises systems, cloud platforms, edge computing environments, and off-site locations. Moreover, the continuous evolution of technologies and applications, including virtual machines, containers, and large-scale data repositories, introduces a dynamic array of data types that require protection.

These organizations not only have to support diverse IT environments but also contend with the exponential growth in data volume that necessitates additional regular management and maintenance. This presents new challenges in terms of data backup and recovery, alongside a host of other expanding demands such as compliance with security regulations, RTOs, RPOs, and ever-tightening budget constraints.

Bacula addresses all of these challenges at once through an agile, contemporary, and modular architecture, which was purposefully designed with open principles to navigate the complexities and extensive data volumes of the HPC environment. Notably, it places IT security at the core of its functionality, integrating a purpose-built security foundation throughout the entire product.

Recognizing that the traditional backup industry’s focus on metering data volume has become unrealistic in the context of growing data volumes, Bacula employs a more sensible and far more cost-effective licensing model based on non-punitive parameters rather than data volume. The inevitable data volume expansion in organizations is thus accommodated.

Bacula Enterprise stands out by enhancing flexibility, automation, and customization options across all aspects of an HPC user’s IT infrastructure, surpassing its competitors in these regards. Simultaneously, Bacula maintains a continuous and integrated security architecture throughout its system, ensuring comprehensive protection over and above that of its peers.

Let’s cover some of these topics in more detail.

Bacula Enterprise: Tape library support

For HPC environments dealing with petabytes of data storage, tape storage can still be the most suitable choice for long-term archiving and meeting RPOs. Tape storage is highly effective when it comes to retention requirements and media preservation. Present-day LTO-08 and LTO-09 tape drives offer impressive specifications, boasting transfer rates of up to 400 MB/sec native, as well as storage capacities of up to 18 TB raw and up to 45 TB in compressed form.

Bacula Enterprise is an excellent choice for tape administrators because it does not license based on data volume and provides distinctive features like ACSLS support. Bacula offers support for tape libraries from leading manufacturers worldwide and encompasses all tape library management operations. Additionally, it supports named user access to ACSLM, tape drive and volume locking in shared ACSLS environments, lock query and management, static tape drive location mapping, and dynamic volume location mapping. This makes it an ideal solution for organizations with substantial tape storage needs in their HPC environments.

Bacula Enterprise: Low Risk implementation processes

IT departments require the seamless integration of robust security measures within their backup and restore systems, minimizing disruptions to operations, all while ensuring comprehensive coverage for innovative technologies like containers and microservices. Achieving this ideal state necessitates an initial integration of the backup system that is exceptionally low risk, involving technology with a minimal learning curve for success.

Bacula’s implementation and configuration processes are remarkably simple and straightforward for a proficient systems administrator with at least a little knowledge of Linux. The system’s high degree of flexibility empowers Bacula to adapt and provide diverse options for interoperability with various other systems. Moreover, Bacula offers the capability to create custom scripts at virtually any level, allowing for precise alignment with the specific requirements of an HPC environment. This ease of use and adaptability minimizes the initial barriers to successful integration.

Traditional backup and recovery vendors often struggle to effectively handle the wide array of systems and data types prevalent in today’s and future IT environments. This limitation becomes increasingly problematic when new requirements emerge, and IT departments are confronted with the prospect of costly custom work, high contracting expenses, uncertainty about the security of proposed solutions, or extended delays in receiving support, which can be six to twelve months down the road.

Bacula takes a slightly different approach. Its software is designed with a modern, modular architecture, making it especially scalable and well-suited for protecting mission-critical IT environments with substantial data volume. It also adheres to open standards, and other de facto standards that are emerging in high performance storage.  Not only is Bacula’s licensing model decoupled from data volume, it is available via an annual subscription, helping HPC environments and research organizations to firstly drive costs down, then accurately forecast future costs while eliminating the concern over uncertainties associated with data growth in their backup and recovery software strategy.

Bacula recognizes that data growth is an inherent aspect of HPC environments and, as such, employs a sustainable and more equitable licensing model based on environments rather than data volume. This typically results in considerably lower costs and reduced risks for end-users. Simultaneously, Bacula’s security architecture maintains a continuous and integrated presence throughout its system, significantly reducing security risks when compared to other solutions.

Bacula Enterprise: Security concerns

In today’s landscape, the prevalence of ransomware and other malware incidents demands an exceptionally high level of security integration within backup and recovery systems. For HPC, its increasing connectivity represents a greater risk, primarily stemming from cyber threats capable of exploiting vulnerabilities in technology to compromise the integrity of networks, systems, and data.

Bacula acknowledges that cybersecurity is poised to be one of the paramount concerns for large organizations. This is further exacerbated by their evolving and increasingly complex demands, driven by IT developments and volume of data. In sync with this transformation, regulations, and policies related to data retention are becoming more defined and stringent, necessitating increased levels of compliance from these systems.

Consequently, HPC users are recognizing the need to enhance their systems’ security, and IT managers are actively seeking effective solutions to meet their organization’s cybersecurity requirements. This includes adhering to guidelines like Security Technical Implementation Guides (STIGs) and Security Requirements Guides (SRGs), as well as industry best practices to ensure robust cybersecurity in an ever-evolving technological landscape.

Bacula’s capabilities in this regard are unusually high among backup vendors, offering FIPS 140-2 compliance, automatic TLS usage for all communications within the network, Windows EFS support, multiple data encryption standards, file verification, CRAM-MD5 password authentication, multi-factor authentication, and many others.

Bacula’s robust resilience to malware attacks is based on its superior security architecture, marked by a noteworthy feature: the client remains unaware of the storage targets and lacks credentials for accessing them. This security architecture is fortified by several core elements designed to enhance its security, such as:

• Client Isolation: Clients are not privy to the knowledge of storage targets and lack the necessary credentials for accessing them. This isolation ensures that clients cannot directly interact with or compromise the storage system.
• Dedicated Systems: Both the storage and Storage Daemon (SD host) are dedicated systems, meticulously secured to permit only Bacula-related traffic and administrative access. This strict access control minimizes the risk of unauthorized access or infiltration.Encryption is also available on both sides.
• Dedicated Director: Bacula’s “Director,” the core management module, operates on a dedicated system with similarly restrictive access. This isolation ensures that the Director can maintain control over all Bacula-related activities and interactions.
• Controlled Access: The Bacula Director takes the initiative in all activities and generates one-time access credentials for clients and Storage Daemons. These credentials only allow for Bacula-related actions, preventing unauthorized access or tampering.
• No Direct Client Access to Storage: Bacula Enterprise does not facilitate direct client access to the storage system. This design is not part of the protocol, ensuring that even a compromised client cannot access, read, overwrite, modify, or delete any backup data. This adds an additional layer of security to protect backup data.
• High degree of interoperability with immutable storage technology, including practically every tape manufacturer in the world. This takes heightened relevance regarding physical air-gapping (see further below).

These security measures collectively create a fortified architecture that significantly reduces the risk of malware attacks and unauthorized access, ensuring the integrity and confidentiality of stored data.

Bacula Enterprise: Bare metal recovery and disaster recovery

Bare-metal recovery is a method that ensures backed-up data is in a format that enables an organization to restore a computer system from a completely clean slate, without any prerequisites regarding previously installed software or operating systems. Typically, the backed-up dataset includes the essential components, such as the operating system, applications, and data, necessary to rebuild or restore the system – even to an entirely different piece of hardware, if needed. This capability for highly autonomous system recovery is strongly recommended, especially for government research, defense, and related agencies.

Bacula Systems offers a comprehensive, fast, bare-metal recovery tool available for both Linux and Windows Server environments. This tool empowers research organizations to execute secure and reliable disaster recovery using Bacula Enterprise. With bare-metal backup, an organization can swiftly and safely restore its critical systems, ensuring minimal downtime and data loss in the event of a disaster or system failure.

Bare-metal recovery can be achieved through various methods. Many enterprises opt for a straightforward approach, which involves deploying a standard image, provisioning software, and then restoring data and user preferences. In these instances, all data is often stored remotely, and the system itself holds less significance. However, this approach may not be practical in many cases, and the ability to completely restore a machine to a specific point in time is a crucial function of disaster recovery implementations.

This capability to restore a computer, even one affected by ransomware encryption, to a recent point in time, including all locally stored user data, can be a vital component of a comprehensive defense strategy. This method can also be applied to virtualized systems, although there are usually more preferred options available at the hypervisor level. Bacula can leverage these options for system-level recovery, streamlining the recovery process for virtualized environments.

Bacula Enterprise: Air gapping

An “air-gapped” computer or network is one that lacks any network interfaces, be they wired or wireless, connected to external networks. Consequently, the organization’s data is kept offline and inaccessible from outside sources. To transfer data between the air-gapped system and the external world, it becomes necessary to write data onto a physical medium and physically transport it between computers. Many organizations incorporate air-gapping as a crucial component of their security strategy.

Bacula Enterprise possesses notable strengths in operating independently and autonomously while also offering the ability to write data to a wide array of storage media. This enhances the options available to IT departments over time for the implementation of air-gapping as part of their security practices.

Bacula’s software architecture is notably streamlined and lightweight, with easily met dependencies. It is designed for manual interaction as well. For instance, in exceptional or critical situations, Bacula can be operated and restored onsite using various methods, including delivering instructions verbally over the telephone if required. This allows for localized, standalone operation without the necessity of networked communication.

Furthermore, the capability to operate independently and standalone is crucial for fostering trust in business relationships. Bacula Systems places a high value on the trust it maintains with its customers and refrains from intruding into an organization’s privacy. For instance, Bacula Systems does not perform internal audits of its customers and respects the privacy of its users. This commitment to privacy and trust is a key aspect of Bacula’s approach to customer relationships.

Bacula Enterprise: Vendor lock-in

Mitigating vendor lock-in can be a challenging task, but there are strategies to reduce its impact. One key approach is to use technologies that are as “open” as possible, promoting flexibility and interoperability. Container technologies offer a prime example of this. Containers enable portability by isolating applications from their underlying environment, allowing organizations to move containers across different locations with the assurance that their applications will function consistently, reducing concerns about vendor lock-in.

Bacula supports this approach by offering backup and recovery capabilities at the container level. It provides advanced backup and recovery functionality for both containers and Kubernetes clusters, enhancing the flexibility and portability of containerized applications.

A significant portion of Bacula’s code is either open-source or based on open-source code. Furthermore, Bacula’s architecture largely avoids proprietary standards, instead adhering to open standards and demonstrating a modular and flexible design. This commitment to open source principles and open standards substantially contributes to mitigating vendor lock-in concerns, giving organizations greater control over their data and systems.

Bacula Enterprise: Scalability

A robust backup and recovery system must be capable of scaling to meet diverse operational requirements across various domains, including business systems, command and control systems, embedded and weapon systems, intelligence analysis systems, autonomous systems, assisted human operations, and more. While this scalability is a valuable assumption, there are only a limited number of backup and restore software solutions that can genuinely meet these expansive needs.

Incompatibilities among technologies, such as databases, storage solutions, and network types, can sometimes lead IT departments to a situation where they must employ multiple backup systems from different vendors to cover their entire IT environment. This is not an approach that Bacula recommends. Instead, the preferred strategy is to find a unified solution that can comprehensively address the backup and recovery needs of the entire IT environment through a single platform.

Some HPC users operate application systems with geographically distributed operating regions, spanning national or international borders. It is crucial for a backup and recovery system to effectively support such models. This includes offering flexible interfaces and user interfaces to accommodate various forms of scaling. Scalability also requires that the backup and recovery solution deliver exceptional stability and performance when operating at scale. Bacula, for instance, is capable of scaling to accommodate many thousands of servers, even in mission-critical scenarios, and it provides a wide array of user interfaces, both command-line, and GUI, which can be configured to offer localized or centralized control as needed. This scalability and adaptability make Bacula well-suited to meet the demands of diverse IT environments.

Conclusion

Bacula Enterprise is intentionally designed to drive positive transformation within HPC IT infrastructures. Its exceptional compatibility minimizes obstacles (for example, it supports over 33 different operating systems versions, and an especially large range of file-system types), and its modularity and flexibility enhance adaptability, expediting the introduction of new capabilities.

In an environment where there are plans for new policies, processes, security levels, with economic and technical shifts — or even when such changes are already underway — Bacula’s adaptability and resilience empower IT leaders to future-proof the backup and recovery component of their strategy. Simultaneously, it leverages the significantly reduced risk inherent in Bacula’s architecture for new deployments.

Bacula’s approach enables organizations using HPC to safeguard a broader range of environments with heightened security, at a much faster pace, and with lower associated risk than ever before.