Data Engineer - HO - HEO

The Home Office works to build a safe, fair and prosperous UK. We achieve this through our work on counter-terrorism, policing, fire, crime, drugs policy, immigration and passports. 

Home Office Digital, Data and Technology designs, builds and develops services for the rest of the department and for government. Every year our systems support up to 3 million visa applications, checks on 100 million border crossings, up to 8 million passport applications and deliver 140 million police checks on people, vehicles and property.

Within DDaT, the Data Services and Analytics (DSA) team find innovative uses for Home Office data and contains one of the most advanced data analytics centres in Government. We strive to lead the way in appropriate and proportionate use of data and analytics to deliver joined-up, intelligent services that unlock the value from data and improve decision making, policy and operations. 

What this means is a daily opportunity to work with teams across the Home Office’s areas of responsibility, from borders, visas and passports, to policing and national security, with some of the largest and most varied datasets and in support of critical national initiatives.

Data Services and Analytics consist of technical and non-technical professionals from a wide variety of backgrounds, we are the hub of data expertise within the department. We support our teams with structured learning, both formal and on-the-job, mentoring and feedback. 

We embrace diversity and inclusion to ensure we represent the public we serve, and we are passionate about fair treatment and the wellbeing of our colleagues as part of our ambitions to be a brilliant Civil Service. 

By joining Data Services and Analytics now, you can help drive forward our aim to become recognised as the leading provider of data insight services in Government.

Job description

As a Data Engineer, you will identify, build, modify, manage and deliver new data pipelines and develop new data services to query, enrich and match datasets making use of existing data services. You will collaborate with the Business Analyst, Data Acquisition, Analytics and Infrastructure teams to deliver the data strategy through the design and implementation of technology systems. 

You will support Senior Data Engineers in analysing problems and data analysis issues, looking for underlying causes and assisting in developing solutions. You will also be responsible for delivering data solutions in accordance with agreed organisational standards that ensure services are resilient, scalable and future-proof. 

You will work closely with the Data Services Science community, who will set the scope of your data engineering work through their research of user and business needs and by their designing data pipelines through writing user stories.  

Location 

The current building location for this role is: 

 2 Ruskin Square – Croydon, Soapworks - Manchester, Vulcan House - Sheffield, regional office.

Travel

Travel for this role will not be required; However, very infrequent visits to the office in London may be encouraged to foster team connection and collaboration, especially for candidates based nationally. Any required travel would be limited to occasional trips, with no overseas travel anticipated, and overnight stays unlikely. We understand that caring responsibilities and other commitments outside of work are important considerations for our candidates and aim to accommodate those needs wherever possible.  

Working Pattern

This role is available on a full-time basis with the option of compressed hours working.  

Key responsibilities will be: 

Assisting in the development of new data models and data enrichment processes, working collaboratively with the Analytics team to build data streaming systems. 

• Designing, building and testing data products based on feeds from multiple systems using a range of different storage technologies and/or access methods.
• Applying the concepts and principles of data modelling to produce, maintain and update relevant data models for specific business needs.
• Assisting in designing, writing, iterating and optimising code from prototype to production-ready.
• Undertaking data profiling and source system analysis and presenting clear insights to colleagues to support the end use of the data.
• Assisting in the successful delivery of completed data loads for customers, Data Engineers and Data Scientists, putting in place remedies and troubleshooting when required and assisting in the development of new data load programmes.

You will also be expected to carry out the following day to day activities:

• Supporting change and bug fix for complex data products in response to business and product manager needs.
• Support the development of requirements from full development, test and through to deployment lifecycles, while maintaining the schedule of work.
• Designing, coding, testing, correcting and documenting simple programs or scripts under the direction of others.
• Support with the development of methods to monitor and improve data quality in source systems using feedback from the Analytics team.
• Undertaking changes to existing routines following change control procedures and then liaising with team members and other teams to communicate delivery and impact of change.
• Work with data privacy and information security staff to assure that security and privacy requirements are identified and addressed in your solutions.
• Work closely with the Data Engineering community to promote challenge from, collaborate with, and ensure an agile approach to working is being adopted. You will also mentor new Data Engineers and Associate Data Engineers.

Person specification

Essential Criteria

• Using tools such as industry standard ETL tools, network databases and scheduling and orchestration tools.
• Knowledge of SQL is a must have skill and candidates must be capable of writing performant and complex database queries using databases such as Postgres, AWS Redshift, AWS Athena or Oracle.
• Demonstrable experience of having written code using either Python or Java and ideally both & an understanding of Linux & data analytics approaches.
• Effectively managing and communicating with a variety of stakeholders by translating technical concepts into non-technical language.
• Managing the development and delivery of technical products.

• Understanding Cloud Data technologies, solutions and future Cloud Data Strategies. An appreciation of core basic Azure and AWS components such as S3, EC2, Redshift, Glue, Athena, DMS, MSK, RDS Kubernetes, Kafka Lambda and Redshift Spectrum is required (in depth knowledge and operational experience is not required).

• Applying data development / engineering techniques – both in theory and practice including data mapping and modelling or a strong aptitude to learn.

All of these requirements are important for the job, however candidates are not expected to be expert in all of them. Just give as much evidence as you can for each and make sure you cover ‘how’ you demonstrated the requirements and the impact of your work, as well as ‘what’ you did. 

The criteria listed above are reflective of the Home Office DDaT Profession Skills and Competency Model (based on the industry standard SFIA framework). SFIA 8 — English (sfia-online.org 

Please see below for the relevant skills required for your role and refer to the “Skills library” attachment for descriptions.

Strategy and Architecture: 

• Business Strategy and Planning

Innovation (INOV) – Level 3

• Technical Strategy and Planning

Data Management (DATM) – Level 2

• Technical Strategy and Planning

Methods and Tools (METL) – Level 2

Development and Implementation:

• Systems Development

Data Modelling and Design (DTAN) – level 3

Database Design (DBDS) – level 2 

Development and Implementation

•    Data and Analytics

Data Engineering (DENG) – level 3 

Desirable Criteria 

Ideally you will also have the following skills or some experience in:

Working with Big Data tools and data stores.

• Using modern / open source programming languages and tools – including to deliver data development / engineering products for example Python, SQL, Gitlab, Java, Docker or Drone. Experience of Java would be highly advantageous, as would knowledge of streamed data technologies, particularly Kafka.
• Knowledge of containerisation, Docker would be advantageous.
• Some experience of existing Home Office and OGD data sets would be advantageous.

Desirable Qualifications

• A numerate degree or equivalent experience.
• An Agile Foundation.
• ITIL v3 or v4 Foundation.

Technical skills

We'll assess you against these technical skills during the selection process:

• Strategy and Architecture > Business Strategy and Planning > Innovation level 2
• Strategy and Architecture > Technical Strategy and Planning > Data Managements level 2
• Strategy and Architecture > Technical Strategy and Planning > Methods and Tools level 2
• Development and Implementation > Systems Development > Data Modelling and Design Level 3
• Development and Implementation > Systems Development > Database Design level 2
• Development and Implementation > Data and Analytics > Data Engineering level 3