Heirloom Carbon Overcomes Data Loss and Boosts Efficiency with AWS

DevOps

High – Performance Computing

  • 22 September 2023
Share this post
AWS Funding Secured by Cloud 303
  • Jump Start
  • Well-Architected
  • MAP 2.0

About the Customer

Heirloom Carbon is a visionary enterprise at the forefront of atmospheric carbon reduction technologies. With a scalable approach to Direct Air Capture (DAC), the company has developed a unique solution that utilizes Earth-abundant minerals, powered by renewable energy sources, to remove CO2 from the atmosphere on a gigatonne scale. Their journey towards mitigating global carbon emissions hit a critical roadblock, however, when they faced massive data loss and operational inefficiencies.

Summary

Heirloom Carbon is a company dedicated to developing technology to reduce atmospheric carbon levels worldwide. By developing scalable direct air capture (DAC) technology, paired with engineering geological carbon storage, Heirloom Carbon offers a path to removing ambient CO2 at the gigatonne scale. They have developed a solution that utilizes Earth-abundant minerals, namely calcium carbonate (CaCO3 ), and by powering the process with renewable energy they are able to capture carbon dioxide (CO2 ) directly from air while maintaining a carbon negative environmental impact.

As a pioneering company in the field of carbon capture and storage, Heirloom Carbon's technology demands significant computing power and robust data storage solutions to model and manage the complexity of their processes. As the company's ambitions grew, it became evident that their existing infrastructure was unable to keep up with their rapidly evolving needs. This is where the AWS Migration Acceleration Program (MAP) came into play.

The AWS MAP provided a comprehensive framework to facilitate a smooth migration of Heirloom Carbon's complex computational workloads from their existing setup to the AWS cloud. The three-phase methodology of the AWS MAP - assessment, mobilization, and migration and modernization, played a crucial role in this transition.

Problem Statement

The air that loomed over Heirloom Carbon was filled with dread towards the tail-end of 2022 when their worst fears were realized. A significant portion of the company’s data, vital for fluid flow simulation and heat transfer engineering, was lost during a standard upgrade procedure performed by their prior managed service provider, and all hope was lost when they discovered the crucial data could not be recovered since no backup policy or disaster recover plan existed.

As Heirloom Carbon scrambled to recover from its data loss, the clock ticked away, threatening to set back its efforts to reduce carbon emissions. The company had moved quickly to adopt cloud technology, but its lack of experience with AWS had left them vulnerable to the shortcomings of its previous partner. In the face of this crisis, Heirloom Carbon needed a partner who could not only provide technical expertise but also guide them towards a more secure and scalable infrastructure.

Why Cloud303?

  • Demonstrated Expertise in HPC Cloud303 possesses specialized expertise in HPC, which is crucial for applications that require complex computational processes. This includes genomics sequencing, molecular modeling, and advanced simulations.
  • Robust Infrastructure The infrastructure provided by Cloud303 is tailored to meet the stringent performance, reliability, and scalability needs of HPC. Our team offers a robust ecosystem that can handle large-scale and intricate computations.
  • Exceptional Support and Security Cloud303 offers round-the-clock exceptional support, along with proven security protocols, to ensure that the sensitive data and complex workloads are managed in compliance with industry standards.
  • Proven Track Record Cloud303 has a strong history of successful partnerships within the life sciences industry. Our commitment to excellence, reliability, and client-focused solutions have made us a trusted partner.

Engagement Overview

Cloud303's engagements follow a streamlined five-phase lifecycle: Requirements, Design, Implementation, Testing, and Maintenance. Initially, a comprehensive assessment is conducted through a Well-Architected Review to identify client needs. This is followed by a scoping call to fine-tune the architectural design, upon which a Statement of Work (SoW) is agreed and signed.

The implementation phase kicks in next, closely adhering to the approved designs. Rigorous testing ensures that all components meet the client's specifications and industry standards. Finally, clients have the option to either manage the deployed solutions themselves or to enroll in Cloud303's Managed Services for ongoing maintenance, an option many choose due to their high satisfaction with the services provided.

Solution Provided

MAP Assess

During the assessment phase, Cloud303 - an AWS Premier Consulting Partner - used Migration Acceleration Program (MAP) to help understand Heirloom Carbon's readiness for the migration. It offered a clear view of the costs, potential risks, and the numerous benefits associated with the migration. With MAP, Heirloom Carbon and Cloud303 were able to chart out an effective migration strategy that aligned well with their business objectives.

In the mobilization phase, MAP equipped Heirloom Carbon's team with necessary tools and training to ensure a seamless transition. This included understanding the AWS environment, the implementation of best practices, and hands-on training to manage the new infrastructure effectively.

OpenFOAM and Ansys Fluent: Carbon Capture Technologies

For Heirloom Carbon, the use of OpenFOAM and Ansys Fluent is critical to their goal of reducing carbon dioxide (CO2) in the atmosphere. These software packages enable Heirloom Carbon to simulate the performance of different carbon capture technologies and evaluate their effectiveness in reducing atmospheric carbon levels. Through simulations Heirloom Carbon is able to develop more effective carbon capture technologies to deploy at scale, reducing the carbon footprint of industrial processes.

OpenFOAM and Ansys Fluent are widely used computational fluid dynamics (CFD) software packages commonly used to simulate fluid flow and heat transfer in engineering and environmental applications. By deploying these software packages on AWS, it allows researchers and engineers at Heirloom Carbon to model and analyze complex systems difficult to study using traditional experimental methods.

Slurm is a workload manager that is used to manage and schedule jobs on a compute cluster and is particularly beneficial when used in conjunction with OpenFOAM and Ansys Fluent jobs. Slurm allows efficient compute resource management for jobs that are computationally intensive, and requiring significant memory and processing power. Cloud303 engineers implemented Slurm using Amazon EC2 instances to optimize and automate the allocation of AWS resources as needed. This ensures that jobs are processed quickly and efficiently. Additionally, Slurm provides a fair share scheduling algorithm, ensuring that all users get a fair share of compute resources. It prevents any one user from monopolizing resources which could cause delays for others. Ultimately, these improvements result in faster job turnaround times and improved overall efficiency.

Managing HPC with Slurm

The Slurm head node was deployed in a private subnet alongside the compute nodes in a cluster placement group to optimize performance. AWS ParallelClusters were used to provision and manage cluster compute resources such as the Slurm head node, compute queues, and auto-scaling infrastructure.

ParallelCluster is an open-source cluster management tool for deploying and managing High-Performance Computing (HPC) clusters on AWS. AWS ParallelCluster uses a simple text file to model and provision all the resources needed for the HPC application(s) in an automated, secure manner. It supports multiple instance types, job submission queues, and job schedulers like Slurm.

Cloud303 leveraged Amazon FsX for Lustre - a high-performing file system - to ensure Heirloom Carbon’s simulations were executed in a timely and efficient manner while maintaining durability and availability of data. This is a critical aspect of the architectural design: OpenFOAM jobs often involve large and complex simulations that generate a significant amount of data.

Slurm was also configured to manage the powering of nodes when jobs were submitted and shutting off nodes when jobs were complete. An Ansys Fluent license server was hosted on Heirloom Carbon’s corporate network. IP whitelisting controlled access, thus eliminating the need for a site-to-site VPN.

The Slurm head node had Paraview installed to visualize the solution and/or mesh. NICE DCV was used to create Remote Desktop sessions into the SLURM Head Node to use the Paraview GUI. Amazon Simple Storage Service (s3) was used to store job inputs and outputs. All S3 objects utilized a lifecycle policy to transfer infrequently used data into cheaper storage classes. A CLI script was written to submit jobs. This CLI uploaded the required inputs to the S3 bucket before initiating the job.

Cloud303 also provided a notification system to notify users when jobs reached certain key states. This was necessary because of the long duration of OpenFOAM and Ansys Fluent jobs. Cloud303 provided Slurm accounting capability to collect accounting information for every job and job step executed, cluster resource utilization monitoring, multiple users per cluster support with NICE DCV session isolation, and restricted access to all compute resources via OpenVPN, durable log storage, and all using AWS Best Practices. Cloud303 also conducted a Well-Architected Review and provided a Success Criteria Report to Heirloom Carbon.

Engineer Quotes

Our focus was not just on technical implementation but aligning it closely with Heirloom's business objectives. Leveraging AWS MAP, Slurm, and Amazon FsX for Lustre, we were able to dramatically improve Heirloom's computational fluid dynamics capabilities. The end result has been a more efficient, cost-effective system that is helping Heirloom innovate faster in the field of carbon capture technology.

Tim Furlong Principal Solutions Architect, Cloud303

Outcomes

Cloud303's solution allowed Heirloom Carbon to develop a scalable, adaptable application on AWS. No matter where the future takes this environmental research company, the infrastructure Cloud303 built on AWS will be able to handle any processing tasks researchers throw at it. On the slim chance that another data disaster does occur, Heirloom Carbon has peace of mind with a robust disaster recovery solution in place to protect and recover their data.

Post-migration, AWS MAP continued to assist Heirloom Carbon in optimizing their new AWS environment. This led to a significant boost in their computational capabilities and performance, thereby enabling them to model and analyze their carbon capture and storage processes more efficiently.

With the help of the AWS Migration Acceleration Program, Heirloom Carbon not only migrated their computational workloads effectively but also gained substantial knowledge and skills to manage their new AWS environment efficiently. The successful transition has allowed Heirloom Carbon to focus on their core mission - developing scalable and sustainable solutions to reduce atmospheric CO2 levels worldwide.

On The Slim Chance That Another Data Disaster Does Occur, Heirloom Carbon Has Peace Of Mind With A Robust Disaster Recovery Solution In Place To Protect And Recover Their Data.