Data Backup and Recovery in HPC and Big Data

Bacula: Unbeatable in HPC and Super Computing Environments.

For an HPC data backup and disaster recovery strategy, you must adapt your IT environment to meet tomorrow’s security standards, integrate with increasingly diverse applications, data, and file types, and meet compliance requirements – and yet reduce costs at the same time. For example, these factors are more critical in the education, research, and defense sectors. There are very few backup vendors that can seamlessly perform these tasks. But Bacula can – with ease. For example, only Bacula can handle billions of files or perform strongly when high throughput is required. Bacula offers exceptionally high levels of security and performance in HPC environments when compared to other vendors.

“Of those evaluated, Bacula Enterprise was the only product that worked with HPSS out-of-the-box without vendor development, provided multi-user access, had encryption compliant with Federal Information Processing Standards, did not have a capacity-based licensing model, and was available within budget”    NASA

With its Open Source pedigree, Bacula Enterprise perfectly supports Posix compliant filesystems, helping you to avoid vendor lock-in. In addition, every filesystem that can be mounted to a Linux or Windows host can be used, including parallel and clustered filesystems such as Lustre or Quobyte. Today, POSIX file systems are the most common storage systems, providing a wide range of IO functions for applications to use, including byte-level access. However, with the large number of IO functions comes complexity, both for the application and the file system. Bacula helps HPC users significantly reduce complexity by being file system-agnostic. Here is a non-exhaustive list of filesystems that Bacula Systems customers use:

ext 2|3|4
xfs
ReiserFS
BeeGFS
GlusterFS
GFS
GPFS
Hadoop *
HPSS
Lustre
Quobyte

ZFS
Swift *

*requires a Bacula Enterprise module (plugin)

Contact Bacula to find out more regarding its full integration with ZFS and its extremely high performance design. In addition, Bacula has further developed its renowned Deduplication capability with new and more powerful scan and recovery tools.

As a proven HPC backup and recovery solution of exceptionally high performance, Bacula can easily handle vast volumes of data. With the increasing need for HPC solutions and further technological improvements, organizations are focusing on areas such as hybrid HPC solutions. Bacula anticipates technological and innovation improvements in the HPC space will increase, especially in Hybrid Cloud, edge compute, container technologies and security approaches. As IT teams work hard to create a balance between on-premises HPC solutions and cloud, Bacula provides a way to protect and recover these entire environments from a single platform.

Read the Bacula whitepaper that covers Backup and Recovery Considerations for this sector:

Top 10 Whitepaper Highlights

1. IT environment complexity in the research sector
2. Technical & demanding IT environments
3. Meeting RPO’s and RTO’s
4. The need for especially high levels of security
5. Bare metal recovery
6. The need to de-risk implementation
7. Hybrid cloud in the research sector
8. Stand-alone capabilities and “air-gapping”
9. Container technologies in the research sector
10. How NASA benefits from Bacula
11. Avoiding vendor lock-in

“Our new backup solution not only had to offer a simple licence model, but also be able to cope with billions of files and petabytes of data. Bacula proved to be a powerful solution that can efficiently backs up large environments.”  –  University of Ulm

High Levels of Automation for HPC Users

Bacula offers an amazing degree of customization and automation in practically every part of its broad feature-set. Contact Bacula for more details on its high automation levels.

The Broadest Tape Compatibility of any backup vendor

For HPC environments that have petabytes of data to store, tape can still be the best option for long term archival and meeting RPO’s. It’s also very effective regarding retention requirements and media preservation. Today’s LTO-08 and LTO-9 tape drives have impressive specs including transfer rates up to 400 MB/sec. native, and storing up 18 TB of raw and up to 45 TB of compressed capacity. Bacula Enterprise is ideal for tape admins as we never license based on data volume and deliver unique features such as ACSLS support. Bacula supports tape libraries from the world’s top manufacturers and provides all tape library management operations. It also supports named user access to ACSLM, tape drive and volume locking in shared ACSLS environment, lock query and management, static tape drive location mapping and dynamic volume location mapping. Please note: Bacula is an industry leading backup and recovery solution for immutable backup.

Total Fit: Bacula’s Ultra-Broad HPC Compatibility

Bacula’s extensive compatibility with HPC technologies contributes to a positive change in technologies and tools to automate processes and to apply consistent governance across an HPC Federal/National laboratory or research organizations. For example, Bacula offers native integration with technologies such as HPSS,  Hadoop, Docker, Kubernetes, SAP Hana, GPFS, etc. Bacula supports over 33 different operating system versions!

As more HPC-driven research and government agencies shift to using Bacula, a pattern of needs clearly emerges from these organizations, and the reasons why the shift takes place. The Bacula Systems white paper ‘Data Backup and Recovery in Federal and National Research Agencies‘ discusses the considerations and advantages of using FIPS 140-3 compliant Bacula Enterprise as a central data backup and recovery system within research organizations, and how it can – and does – facilitate a specific, yet critical part in these organization’s enterprise-wide approach to digital modernization.

Large Backup File Split

This Bacula feature backs up large numbers of files with an optimal speed and efficiency, splitting the workload in a number of workers (concurrent backup processes).  Bacula’s Large Backup File Split saves significant time in HPC environments when dealing with very large datasets. It can work with any kind of file system, and the backup is automatically managed by Bacula. This is applicable to any file-server or NAS use-case.

Another attraction of Bacula’s automation of this process is its ability to reduce the possibility of human error. It is also fully dynamic, meaning no configuration update process is necessary, regardless of any changes in the structure or size of the target dataset.

This tool can especially help where there are many users, resulting in a complex folder structure and lots of different files. This kind of file structure may literally be being used by everybody (e.g. public/private, internal company users, shared data situations, etc.). More details are available in Bacula’s HPC white paper.