Atlanta SC24 Reflections
Now that the dust has settled, we can look back on the experience that was SC24 in Atlanta. We arrived
Case Study
How Red Oak’s consultancy paved the way for the Met Office’s new direction
The Customer
The Met Office, synonymous with providing the UK public with weather forecasts over the years, is in fact, so much more than that.
Founded in 1854, the Met Office has pioneered the science of meteorology and its application, earning a reputation for continually pushing the boundaries of science, technology and innovation in meeting the demands of today and the future.
In addition to being the nation’s weather and climate agency, it provides aviation meteorology forecasting for no less than two thirds of all flights from across the globe’s airlines, daily, so if a flight needs to travel from New York to London, it won’t be able to take off without a forecast from the Met Office.
Furthermore, it collects and analyses global climate and scientific data for energy providers, renewables operators for wind farm placements, utilities, and helps government, businesses, emergency responders and the public to make informed decisions connected with weather.
The Challenge
None of the work which the Met Office does can be achieved without High Performance Computing (HPC), its supercomputer being one of the largest in the world, capable of 14,000 trillion arithmetic operations a second – which equates to 215 billion global weather observations every single day.
OBS data (observations) from aircraft, ships, weather stations and outposts from far across the globe flows into the Met Office each day, amounting to billions of observations. And with numerical weather predictions (NWP) running different simulations across the globe – global, regional (Europe), UK-specific – right down literally to 100m stretches of road. The task of such a level of data harvesting as this is vast, and it was soon apparent that the on-premises HPC was coming towards the end of life.
As such, the Met Office needed a clearer view of HPC market trends and a greater understanding of its data archive and, vitally, how they were using it. Moreover, the Met Office was gradually running out of power, and space – all of which was becoming a big issue.
Effective stakeholder engagement was essential to the process of securing the necessary buy-in from across the Met Office’s decision base, from the top down, helping with evaluations and running projects. In this way, Red Oak was able to help the Met Office get to a point where everyone was on the same page, understood their requirements, and shared a common, singular goal across the whole organisation.
Understanding the scale of the Met Office, the types of data, workflows and applications, Red Oak applied its intimate knowledge of the HPC market towards what could potentially be the right investment. Two options were available; stay with an on-premises cluster, and have it located off-site in a data centre which could be scaled up to meet demand over the years, or think outside the box and look at the Cloud.
And that’s where Red Oak Consulting came in and advised on how to look at things differently.
What did Red Oak Consulting do?
Red Oak Consulting’s senior consultancy team worked alongside the Met Office, including principal fellow for technology, Richard Lawrence, to help them understand what transformation would mean for the business.
It was vital they understood how a significant shift in direction would affect them and what potential advantages and pitfalls might exist in taking a different approach.
In essence, to think differently about what they were wanting to achieve and to realise that the Met Office couldn’t just do what they’d always done, rather than just making continued financial investment in the existing paradigm.
At this point in the project, Red Oak conducted a detailed analysis of the data archive to understand its actual usage versus perceived usage. Additionally, Red Oak offered specialised resources and insights for the data archive, which stores vast amounts of weather and climate data.
This strategic consultation enabled the Met Office to articulate its vision more effectively, attracting more bids and fostering innovation from potential suppliers, including Cloud providers like Microsoft.
Furthermore, Red Oak’s consultants have provided specific guidance on optimising use of the Met Office’s supercomputer and efficiently delivering products to customers, leveraging niche skills and tools.
Finally, to enable the Met Office to scale up to six times the computational power without having to build and provide operational support for a much bigger data centre, Red Oak enabled a smooth transition to the Cloud.
We've been involved across the full spectrum with the Met Office with horizon scanning and helping them pick the technologies that were going to be coming through for their next procurement. Likewise, with engaging with the vendors and helping with procurement activity.
“And it’s also about data management, how you archive this data and manage data growth, given they produce maybe hundreds of terabytes every day. So, it’s about a lot more than just predicting the weather, and we worked with them to help liaise with the Government on socio economic benefits, food security, energy security and predicting the impact of climate change over the next, 10, 20 or 50 years. The Met Office is one of these truly competent organisations around HPC and has a very long pedigree of competency.”
Owen Thomas, Red Oak Consulting
The Results
A ground-breaking first, with the Met Office making the strategic decision to transition to the Cloud with the aim to fully implement Cloud-native HPC on the Microsoft platform.
Red Oak Consulting’s efforts led to a transformative shift in strategy for the Met Office. As a result, the agency awarded the Cloud contract to Microsoft, making it the first UK government organisation (and one of the pioneers globally) to adopt a Cloud-based HPC system for weather and climate modelling and forecasting.
The new system, which is due to go live later this year (2024), will provide the Met Office with far greater flexibility, scalability, resilience and security, as well as access to Microsoft’s Cloud resources and innovation.
HPC is a critical component for the Met Office. Without it the ability to draw down and analyse the data could not function. For more than 15 years Red Oak Consulting has supported the Met Office with its HPC stability.
Red Oak has been responsible for the single biggest change in our strategic direction. Over the past five years, they spotted a trend in the industry, that wasn't very widespread. At that point they saw a shift in HPC from being traditional on-premises provision to being more in the in the Cloud. Red Oak is Probably the only company really that has the specialisation in this space.”
Richard Lawrence, Principal Fellow for platforms at the Met Office
Get the latest HPC insights delivered straight to your inbox
Sign up to our newsletter to stay up to date with all the latest news and advancements in High-Performance Computing.
Certifications
Latest News
SC24 A Rookie Robot Reflects
Lights, Camera, Clusters! Hey there, fellow computer enthusiasts! Roco here, everyone’s favourite artificially intelligent robot (and modesty chip malfunctioning… malfunctioning…).
My First Supercomputing Adventure:
A Rookie Robot’s Guide to SC24 Hey there! Roco here, Red Oak Consulting’s newest (and cutest, right?) team member! I
Revolutionising the Automotive Industry:
The Power of High-Performance Computing The automotive industry is in the midst of a transformation, driven by world-leading technology, evolving consumer
