Lead Data Engineer (Data and Infrastructure)

The Data, Technology and Analytics (DaTA) unit is an established multidisciplinary team that brings together data engineers, data scientists, technologists, behavioural scientists, and other experts. You will join a team of about 20 Data Scientists and Engineers within the wider unit. As a Lead Data Engineer, you will be involved in designing and managing our analytical infrastructure and supporting the delivery of our most technically challenging projects. On one project, you may engage with tech giants like Google, Amazon, and Facebook and help to navigate vast data landscapes and understand how to regulate these digital powerhouses, and in another, you may develop a custom solution in AWS for data scientists to easily run ad-hoc scheduled jobs, or experiment with LLMs (AWS Bedrock and open-source models) and RAG systems as part of our digital transformation strategy.

In this pivotal role, you'll be at the forefront of developing data-driven tools and data pipelines that enable the CMA to analyse and regulate these influential platform businesses, and others. You'll build robust data infrastructure to support critical cases and market studies, putting machine learning models in production that detect anti-competitive behaviours, process the news and consumer complaints for our intelligence pipelines, and generally contribute to foster fair competition and protect consumers.

As a Lead Data Engineer, you'll play a vital role in crafting solutions to these issues, ensuring digital markets remain competitive and beneficial for consumers.