DDaT Data Engineer - MoD - SEO
Government Digital & Data -
Do you want to work at the forefront of UK Defence, supporting operations that are beyond the reach, capability, and expertise of other armed forces?
We are a tri-service organisation that recruits the best of the armed forces and civilians to perform the most rewarding and complex tasks. We are recruiting exceptional individuals to support our activities tackling some of the most interesting and challenging national security issues in government. You will be working with unique, highly sensitive data, not seen elsewhere in defence, to the backdrop of a macro environment that is ever changing.
Some of the challenges you will face include working in mixed military and civilian teams to enable high risk operations by providing timely, accurate and trustworthy data and the development of cutting-edge capabilities.
This position is advertised at 37 hours per week.
Job description
You will be supporting the delivery of data products and capabilities used by our soldiers and marines on discreet military operations promoting the UK’s national interest across the globe. As our capabilities become ever more complex, consuming larger quantities of data, the effective, reliable, and scalable ability to process data into consumable products is essential.
The team in London is a mix of civilian, military personnel and contracted support working in collaboration with other teams across locations in the UK and overseas. We work across security classifications and have direct access to the operational users need.
We work closely with NATO, Five Eyes and other partners. There is opportunity for international travel to support operations within risk tolerance.
Some of the responsibilities you will take on include:
- Implement and automate data flows to connect operational systems and data for analytics to exploitation systems.
- Write scripts to extract, transform and load (ETL) data for ingestion and structure/format data for export around other parts of the business.
- Develop and implement data engineering best practice (source-to-target mapping, code standards, data quality, generating representative datasets etc.)
- Support and mentor other data engineers as well as develop your own skills.
- Setup and administer global policies for data.
Person specification
Essential Criteria
- NiFi, Airflow and Azure Data Factory
- Python and Pandas
- SQL/NoSQL Databases (Neo4j, Elastic, Azure Blob)
- Messaging technologies such as Kafka
- Storage solutions such as Min-io or CEPH
- Experience of agile working
- Kubernetes (AKS and Kubernetes Engine)
- DevSecOps best practices, including source control (Git) and vulnerability scanning
Desirable Criteria
- AWS & Azure solutions for storage and data engineering, particularly tools available on Stack Hub
- Data analysis and visualisation tools such as Palantir, Tableau, PowerBi, TAK
- Familiarity with data governance and lineage practices
Behaviours
We'll assess you against these behaviours during the selection process:
- Seeing the Big Picture
- Delivering at Pace
- Changing and Improving
- Leadership
Technical skills
We'll assess you against these technical skills during the selection process:
- Data development process
- Data integration design
- Programming and build (data engineering)
- Problem resolution (data)
