Lead Data Engineer- Azure- CRM Dynamics

Lead Data Engineer- Azure- CRM Dynamics

Lead Data Engineer with in depth knowledge of Azure Data Services with experience of working with CRM Dynamics is urgently required to support our higher education client to implement data quality processes and data profiling.

The Lead Data Engineer will be responsible for conducting comprehensive data profiling and analysis to gain insights into data quality issues, data distribution, and data patterns. Perform exploratory data analysis to uncover hidden trends and opportunities for improvement.

Lead Data Engineer- Azure- CRM Dynamics- Essential Skills

• In-depth knowledge of Azure data services including, Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Azure SQL Database & Azure Stream Analytics.
• Strong Python or Scala with Spark, PySpark experience
• Experience with relational databases and NoSQL databases
• Significant experience and in-depth knowledge of creating data pipelines and associated design principles, standards, Data modelling concepts, ETL procedures, and all steps of data production process
• Experience with unit and integration testing, and data quality frameworks such as Deequ, Great Expectations or Delta Live
• Experience working with continuous integration
• Good understanding of data modelling (medallion architectures, Kimball architectures), data schemas (Avro Schema, JSON Schema) and data serialisation (Parquet, ORC, Avro).
• Experience in Data Migration, Data Profiling and Data Quality enhancement.
• Understanding of CI/CD process & ways of working including GitCI
• Strong understanding of software development life cycle
• Knowledge of software engineering best practices for the full software development life cycle, including Agile methodologies, coding standards, code reviews, source control management, build processes, testing.
• Expertise in Azure Resource Manager (ARM): Understanding how to deploy, manage, and monitor Azure resources using ARM templates.
• Proficiency in using the Azure Portal and Command-Line Interface for managing Azure resources.
• Knowledge of Azure security features and compliance standards.
• Familiarity with Azure Monitor and Azure Log Analytics for monitoring and logging.

Lead Data Engineer- Azure- CRM Dynamics- Desirable Skills

• Strong Python or Scala with Spark, PySpark experience
• Experience of working within CRM Dynamics Marketing & Sales or within a Higher Education environment.
• Experience of supporting specific Higher education systems within the complex of enterprise system integration.
• Ability to utilise advanced data wrangling techniques/tools (Power Query, Dataflows & Data Wrangling using Python)

This role is inside IR35 and is hybrid working with 1-2 days per week on site in central London.

6 month initial contract with scope to extend.

InterQuest Group is acting as an employment agency for this vacancy. InterQuest Group is an equal opportunities employer and we welcome applications from all suitably qualified persons regardless of age, disability, gender, religion/belief, race, marriage, civil partnership, pregnancy, maternity, sex or sexual orientation. Please make us aware if you require any reasonable adjustments throughout the recruitment process.