Data Engineer

Posted 6 days ago by Be-IT Resourcing
Easy Apply Ending soon

An exciting opportunity has arisen for a Data Engineer to design, build and maintain the data pipeline and data acquisition processes to deliver well structured, efficient and well performing data to drive business analysis, insight and processes. As a Data Engineer you will have an opportunity to influence the design and architecture as you help to develop a Cloud Data platform. You will liaise with Enterprise Architecture and Business areas to understand data assets and ensure these assets can be connected and represented on the platform.

You will promote adoption and adherence to the best practices for developing in Cloud infrastructures via engagement with colleagues across the business but particularly in Data Engineering, Data Governance and CIO. Ultimately you will help develop the platform to enable that the data assets be understood and accessed to maximize their value.

Reports to the Data Engineering Manager or Data Engineering Leads for direction in terms of priorities or support. Support the Data Engineering Manager through delivery of projects/work within deadlines, prompt escalation of risks or issues and ensuring that day to day activities are delivered within SLA's.

  • Building and maintaining the business data Cloud environment and data assets that will drive the BI, Analysis, Regulatory Reporting and Analytics activities for the organisation.
  • Striving for technical excellence. Ensuring that the Cloud infrastructure and data built within EDS is well structured, efficient and extensible.
  • Being proficient in multiple and emerging technologies. AWS/Azure, CI/CD, APIs, ETL, Python and Jenkins are just a few of the skill sets that you'll work with within the area.
  • Using technology in Industry Best Practice ways. Build data flows which take full advantage of the technologies we have.
  • Building safe Data Flows. Ensure that there are fit for purpose validations and controls in our Data Flows in order to evidence control across the area.
  • Automating everything that we do. Work with our EDS colleagues and CIO to ensure that we build robust, automated solutions to drive the best use of Data Engineering time.
  • Measuring everything that we do. Work with Data Operations to ensure that any checks and measures are embedded into our Data Flows to evidence the accuracy and timeliness of our data.
  • Aligning to Data Governance Principles. Work with Data Governance to build, monitor and maintain fit for purpose documentation such as Data Dictionaries and Entity Relationship Diagrams for our databases and data assets.
  • Maintenance of existing Data Flows. Make changes to Data Flows as necessary to ensure they run smoothly or to enhance them as required by the business. Follow well-structured version control methods to ensure that we can roll back to previous versions and can maintain an audit trail of changes made.
  • Building a strong Risk culture within Data Engineering. Identifying operational and strategic risks and ensuring that these are managed and recorded within the Risk Management Framework.

Skills and Experience

  • AWS or Azure Cloud Experience
  • Experience of Data and Analytics platforms using Cloud infrastructure.
  • APIs
  • Real Time Data Flows - Kinesis/Kafka
  • Lambda
  • Github/Git
  • Jenkins or other CI/CD toolsets
  • Cloud Formation
  • Python/Scala/Java
  • Infrastructure as code
  • Spark/Hadoop
  • Data Science
  • Machine Learning
  • AI
  • Data Design
  • Database/Data Solution Design
  • Experience in SQL
  • Experience in Cloud based ETL tools such as Pentaho/Talend an advantage.
  • ETL Skills
  • Agile methodologies
  • Jira/Confluence

Please apply on line for immediate consideration

Reference: 38690075

Bank or payment details should never be provided when applying for a job. For information on how to stay safe in your job search, visit SAFERjobs.

Report this job