Data Engineer

Location: London, Potters Bar, Bristol or Isle of Man (Hybrid working options available)

Job Summary

The Data Engineer is a technical role responsible for designing, developing, and maintaining data pipelines within the IT department. The pipelines will be realised in a modern lake environment and the engineer will collaborate in cross-functional teams to gather requirements and develop the conceptual data models. This role plays a crucial part in driving data-driven decision-making across the organisation, ensuring data availability, quality, and accessibility for various business needs.

What you'll do

Data Pipeline Development

• Design, model, develop and maintain data pipelines to ingest, store, process, and present data.
• Ensure data quality, accuracy, and consistency.
• Collaborate with data architects to ensure data pipelines align with the overall data architecture strategy.

Data Transformation and Integration

• Perform data transformation tasks, including data cleansing, enrichment, and aggregation, to prepare data for analytics and reporting.
• Integrate data from structured and unstructured sources, ensuring compatibility and alignment with data models and business requirements.
• Automate data transformation processes to improve efficiency.

Data Quality Assurance

• Implement and maintain data quality checks and validation processes to identify and resolve data anomalies and errors.
• Monitor data pipelines for data quality issues and implement data quality improvements.
• Collaborate with business stakeholders to define data quality requirements.

Data Modelling and Schema Design

• Collaborate with data architects and data scientists to design and implement data models, schemas, and structures.
• Ensure that data models support business reporting and analytics needs while optimizing query performance.
• Maintain data dictionaries and metadata to document data structures and relationships.

Performance Optimization

• Optimize data storage, retrieval, and query performance by implementing indexing, partitioning, and caching strategies.
• Monitor data processing performance and address bottlenecks as they arise.
• Stay updated with best practices in data processing performance tuning.

Documentation and Knowledge Sharing

• Create and maintain documentation for data pipelines, data transformation processes, and data integration procedures.
• Foster a culture of knowledge sharing within the data engineering team and across the organization.

Collaboration and Stakeholder Engagement

• Collaborate effectively with cross-functional teams, data stakeholders, and business units to understand data requirements and deliver data solutions that meet business needs.
• Communicate technical concepts and data solutions to non-technical stakeholders in a clear and understandable manner.

Must Have Skills/Experience

Essential:

• Commercial experience of DataBricks, SQL, Python, Power BI
• Experience in data engineering, including designing and developing data pipelines for retrieval / ingestion / presentation / semantics in an Azure environment.
• Effective communication and collaboration skills to work with cross-functional teams and gather data requirements.
• Skills in data modelling (both structured and unstructured data)

Desirable:

• Power Automate
• Data optimisation for performance, scalability, and efficiency.
• Experience in a financial corporation