Lead Data Engineer - HO - G7
Government Digital & Data -
Digital Data and Technology (DDaT) enables the Home Office to keep the UK safe and secure. We design and build the services that help people apply for visas or passports; support policing and counter-terrorism operations; and protect the UK’s borders.
Data Services and Analytics (DSA) is the part of DDaT which leads the build of data products to integrate data from multiple sources, and then deliver products for business intelligence, reporting and data science. DSA has the ambition to be one of the leading providers of insight services in UK Government and is currently growing significantly with over 300 colleagues, providing services to variety of functions within the Home Office.
In this role, you will identify and deliver new data sources and develop ETL processes, collaborate with the Data Acquisition, Analytics and Infrastructure teams to deliver the data strategy through the design and implementation of technology systems.
We value diversity and provide an open, inclusive, and supportive environment to help you do your best work.
The successful candidate will either be located in Croydon, Liverpool, Manchester, Sheffield or Glasgow. Under Home Office hybrid working practices there is an expectation that you will attend the office for a minimum of 60% of your working hours.
Job description
There are two types of Lead Data Engineer positions in the Data Services and Analytics (DSA) team.
There are several positions to lead a team of data engineers working on complex data pipelines, ingesting and transforming data from/to a variety of sources (email, csv, ODBC/JDBC, JSON, XML, Excel, Avro, Parquet). Data pipelines are primarily developed in AWS using a variety of tools and languages including AWS S3, AWS Athena, AWS Redshift, AWS Glue, python and java (Docker / Spring).
There is one position leading the data engineering development for a team of data engineers working on a legacy Oracle data platform and delivering data integration and data retrieval services using PL-SQL. A complex transformation programme is delivering the change of people, process and technology to redesign and transform the processing of the Oracle data outputs onto the strategic data platform. This role would ideally have experience of Oracle and PL-SQL, with the aptitude to move to cloud native tooling such as AWS Athena, python, S3. However, Oracle experience is not essential for this position as the key priority is to manage the people change to ensure the team stability and business continuity throughout the transformation.
As a lead Data Engineer, you will showcase a true passion for data and exhibit expertise in the following areas:
- Manage data sources effectively to improve data quality and make them suitable for analysis.
- Drive and develop new data models and ETL processes, working collaboratively with the analytics team and mapping data to an appropriate matching data model.
- Utilise essential key stakeholder management skills to effectively manage expectations, actively listen and interpret business needs, communicate progress, and navigate challenging conversations, all while fostering a service orientated culture around data.
- Managing the performance of suppliers and the data team to improve capability, as well as creating the roadmap which balances immediate user needs with long-term investment.
- Developing processes to balance data availability, capacity and data archiving, while continually improving processes and efficiency in the acquisition and ingestion of data.
- Identifying data issues in data pipelines and managing their resolution.
- In one of the positions the successful candidate will be responsible for upskilling staff to transition to a new platform. Experience with Oracle is a plus. They must also have leadership experience, including coaching and professional development skills.
Person specification
Essential criteria
As a Lead Data Engineer you’ll have a demonstrable passion for Data, with the following skills or some experience in:
- Effectively leading and managing teams, helping them deliver technical products and projects in data analysis, solution design and end user reporting.
- Automate IT structures by advising on best practice IT programming and automation techniques and tools such as GIThub, GITLab or other source control software, Python.
- You will lead the implementation and strategic use of Cloud Data technologies, solutions, and future Cloud Data strategies particularly in the context of data warehousing and data lake architectures. This includes expertise in AWS services such as Redshift, Athena, EC2, IAM, AWS Lambda and CloudWatch
- Working with stakeholders to gather requirements and deliver findings.
- Presenting complex ideas and recommendations that inform decisions in a clear way to technical and non-technical audiences and senior customers.
- Using tools such as industry standard ETL tools, network databases and scheduling and orchestration SQL tools.
- Strong analytical and problem-solving skills troubleshooting data issues and optimising data processes.
SFIA capability framework
Skills for the Information Age (SFIA) is the technical framework that sets the standard capability and development of all Data Engineering levels in the Home Office. This is a link to the capability framework: All skills A - Z English (sfia-online.org)
We use set SFIA technical skills to form our interview questions and we will assess you against these technical skills during the selection process.
SFIA levels of responsibility – Use the SFIA Levels of responsibility to understand what would be expected for each Technical Skill listed below.
SFIA Technical Skills
The essential technical skills listed above are reflective of the Home Office Government Digital and Data Profession Career Framework. Please see below for the relevant skills required for your role.
