Job Description
Who are we looking for?
- We require a data engineer to join our global team to support an insurance client.
- It requires someone who understands the concepts of managing databases that scale (to over many terabytes and whose data sets can exceed billions of new rows incremented per day) along with a developer approach to using GIT and continuous automation and testing to deliver a high-quality data definition of the truth.
- The role requires the person to work with technical staff and customers and be confident applying technical requirements and advising on the strengths and weaknesses and limitations of Azure Big Data technology.
- Someone who can describe and summarize Azure Data Factory, Databricks, python, data frames, YAML, a GIT Merge Request or contrast an OLTP database against a warehouse columnar store would be off to a flying start.
- We use all the above to deliver great quality data, on time, reliably to allow our customers to govern and manage their data to make great data driven business decisions.
Technical Skills
Essential
- A background in SQL and principles of data warehousing
- Experience in data modelling and ELT
- Analytical and Structured approach to work
- Python
- Azure Data Factory And / Or Databricks
- A team player able to perform peer code reviews and provide constructive mentoring
Desirable
- 3 years’ experience within Data & Analytics
- Experience of ETL design and dev, including Test-Driven development
- Deep understanding of relational as well as NoSQL data stores, methods, and approaches (Kimball star and Data Vault patterns, dimensional modelling)
- GIT experience
- Terraform
- Azure Synapse
- A track record of achievement and recognition within the IT industry as evidenced by significant promotion since leaving University
- A strong academic track record including the achievement of a good honors degree or equivalent
- Certification in Microsoft Azure cloud technology
Summary
ADF Databricks Data Engineer
Process Skills
- Understand and embrace the hybrid Azure Data Factory (ADF), Databricks Delta Lake and Azure Synapse, Power BI Microsoft tech solution
- Work as part of the application monitoring and validation team (AM)
- Work as part of the team to provide Incident Management (triage, comms, and simple critical fixes to adhere to SLA)
- Work as part of the team to maintain an up-to-date Support Status manifest showing the status of open issues and pipeline incidents
- Review and resolve data pipeline failures and data test failures where simple and timely to do so
- Work as part of the team to Maintain and improve the existing support documentation (run books, known issues and data environment)
- Work as part of the team to provide Environment Support (standup of new environments, patching of DEV, SIT, UAT and Prod existing environments)
- Maintain and improve the existing monitoring and data quality tests to improve observability
- Collaborate continuous improvement suggestions to improve reliability, quality, and freshness of the data. Where possible collate these in to proven improvements to the other teams
- Collate and implement automation of manual processes. Where these require more than a man week then package these improvement suggestions for other teams.
- Work with the Enhancement and Development Teams (AME and AD) to review Release Candidate changes against successful acceptance testing to adhere to the customer’s ITIL change management process.
- Perform technical validation, data load tests and promotion of release candidates.
- Provide feedback of potential issues, logic bugs and performance improvements back to the supplying team.
- Work as part of the team to implement the efficiencies of a DataOps CI/CD automated deployment for future
Behavioral Skills
- Work with the Enhancement and Development Teams (AME and AD) to review Release Candidate changes against successful acceptance testing to adhere to the customer’s ITIL change management process.
- Perform technical validation, data load tests and promotion of release candidates.
- Provide feedback of potential issues, logic bugs and performance improvements back to the supplying team.
- Work as part of the team to implement the efficiencies of a DataOps CI/CD automated deployment for future
Certification
- Certification in Microsoft Azure cloud technology
- Azure Data Bricks