Data Engineer

Job Description

Job Title

Data Engineer

Reporting Line

Data Engineering and Solutions

Role Type

Permanent

Experience

6+years

Summary

As a Data Engineer you will work on implementing complex data projects with a focus on collecting, parsing, managing, analysing, and visualising large sets of data to turn information into value using multiple platforms. You will work with business analysts and data scientists to understand customer business problems and needs, secure the data supply chain, implement analysis solutions, and visualise outcomes that support improved decision making for a customer. You will understand how to apply technologies to solve big data problems and to develop innovative big data solutions.

Key Accountabilities

• Working with colleagues to understand and implement requirements
• Securing the data supply chain, understanding how data is ingested from different sources and combined / transformed into a single data set.
• Understanding how to analyse, cleanse, join and transform data.
• Implementing designed / specified solutions into the chosen platform
• Working with colleagues to ensure that the on-prem/cloud infrastructure available is capable of meeting the solution requirements.
• Planning, designing, and conducting tests of the implementations, correcting errors and re-testing to achieve an acceptable result.
• Appreciate how to manage the data including security, archiving, structure and storage.

Key skills and Technical Competencies

• Degree level education in Mathematics, Scientific, Computing or Engineering discipline or equivalent experience
• 6+ years of experience at various levels of Data Engineering roles including 3 years in technical lead role managing end to end solution.
• Hands on Experience with Azure Databricks using PYSPARK with ability to do cluster capacity planning and workload optimization.
• Experience in designing solutions using databases and data storage technology using RDBMS (MS SQL Server)
• Experience building and optimizing Big Data data pipelines, architectures and data sets using MS Azure data management and processing components through IaaS/PaaS/SaaS implementation models implemented through custom
• Experience in Azure Data factory.
• Proficiency in Python scripting. Experienced in using different python modules used for data munging
• Be up to date with data processing technology / platforms such as Spark (Databricks)
• Experience in Data Modeling for optimizing the solution performance
• Experienced in Azure DevOps with exposure to CI/CD.
• Good understanding of infrastructure components and their fit in different types of data solutions
• Experience of designing solutions deployed on Microsoft and Linux operating systems
• Experience of working in an agile environment, within a self-organising team.

Behavioural Competencies

We are adopting a winning mindset, aligning around our strategy, and being guided by a clear set of behaviours

PUT SAFETY FIRST

Prioritising the safety of our people and products and supporting each other to speak up.

DO THE RIGHT THING

Supporting a culture of caring and belonging where we listen first, embrace feedback and act with integrity.

KEEP IT SIMPLE

Working together to share and execute ideas and staying adaptable to new ideas and solutions.

MAKE A DIFFERENCE

Thinking about the business impact of our choices and the business outcomes of our decisions and challenging ourselves to deliver excellence and efficiency every day on the things that matter.

As part of our selection process, candidates in certain locations may be asked to complete an online assessment, which can include cognitive and behavioural aptitude testing relevant to the role. If required, full instructions for the next steps will be provided.

Job Category

Digital