Job Description
The Role
We are seeking a motivated Data Engineer to support the development, and maintenance of a data-bricks lake house architecture. This role is critical in enabling the organisation to store, integrate, and analyse large volumes of data from diverse sources, ensuring data is accessible, accurate, and actionable for decision-making.
Responsibilities
Key Responsibilities
- Data Architecture & Engineering
- Contribute to the design, development, unit testing, and deployment of data pipelines for extraction, ingestion, and transformation of data.
- Support the development of scalable data architecture to process and store datasets efficiently.
- Optimisation of data pipelines from internal and external sources using modern data engineering tools and frameworks.
- Ensure their deliverables meet the team standards and best practices.
- Collaboration & Stakeholder Engagement
- Work closely with data scientists, analysts, and business stakeholders to understand data needs and deliver fit-for-purpose datasets and operational data capabilities.
- Collaborate with team members to support applications and resolve technical challenges.
- Create and maintain high-quality documentation
- Data Quality & Automation
- Optimize automated data quality, cleansing, and validation processes within data pipelines.
Qualifications
Required Skills and Experience
- Technical Proficiency
- Experience in data warehousing, data lakes and data integration, transformation, and modelling
- Hands-on experience with Databricks for scalable data processing and Azure SQL for data storage and querying..
- Proficiency in SQL, Python, and modern data pipeline tools (e.g. dbt).
- Data Governance & Quality
- Demonstrate an ability to implement data quality frameworks and ensure data integrity across systems.
- Familiarity with data privacy regulations (e.g., GDPR) and security best practices.
- Security & Compliance
- Follow best practices for data security and privacy, ensuring compliance with internal policies and external regulatory requirements.
- Apply technical knowledge to deliver key projects, including the automation of data deployment, release, and upgrade processes.
- Cross-functional Collaboration
- Experience working in cross-functional teams and collaborating with other engineers.
Preferred Skills and Qualifications
- Databricks or Azure Data Engineer certifications (e.g. Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer professional)
- DBT Analytics Engineering Certification
