Lead Data Engineer (Databricks & Cloud Migration)
£650 per day (Inside IR35)
Initial 6-Month Contract
London – Hybrid Working (3 Days per Week Onsite)
We are partnering with a leading international bank on a major Databricks-led cloud transformation programme. We are looking for a Lead Data Engineer with deep expertise in Databricks, modern data architectures and large-scale cloud migrations to help deliver a next-generation data platform. You'll play a pivotal role in designing and building Data Marts, developing high-performance data pipelines, and migrating critical data assets from legacy Oracle environments into a scalable Databricks Lakehouse architecture that underpins finance, regulatory reporting and advanced analytics across the organisation.
Key Responsibilities
Databricks & Modern Data Platform Engineering
- Lead the design and development of scalable Data Marts using Databricks.
- Build and optimise cloud-native data pipelines using SQL and Python.
- Drive engineering best practices across CI/CD, version control, orchestration and automated deployment.
- Ensure platform performance, scalability, governance and operational excellence.
- Champion modern data engineering standards and architectural best practices.
Oracle to Cloud Migration
- Lead the migration of data assets from legacy Oracle environments to a modern cloud-based platform.
- Define migration strategies, source-to-target mappings and data model redesign activities.
- Oversee the retirement of legacy ETL processes and implementation of cloud-native alternatives.
- Work closely with architects, DBAs and business stakeholders to ensure data quality, integrity and continuity throughout the migration programme.
Leadership & Stakeholder Engagement
- Provide technical leadership, mentoring and guidance to Data Engineers.
- Collaborate with Architecture, DevOps, Analytics, Finance and Risk teams.
- Communicate complex technical concepts effectively to both technical and non-technical stakeholders.
- Conduct design reviews and ensure delivery aligns with engineering standards and enterprise architecture.
Operational Excellence
- Support production releases and change management activities.
- Drive improvements in automation, monitoring, observability and platform reliability.
- Assist with incident resolution and critical release support when required.
- Ensure documentation, audit controls and governance requirements are maintained.
Required Experience
- Proven experience delivering enterprise-scale Data Engineering solutions using Databricks or similar cloud data platforms.
- Strong experience designing and developing modern data architectures and Data Marts.
- Demonstrable experience migrating Oracle or other legacy data platforms into cloud environments.
- Deep understanding of data modelling, ELT/ETL design patterns and data integration principles.
- Advanced SQL development skills.
- Strong Python development experience.
- Experience with CI/CD pipelines, Git-based workflows and DevOps practices.
- Experience leading engineering teams and providing technical mentorship.
- Strong stakeholder management and communication skills.
Desirable Experience
- Banking or financial services experience.
- Experience within highly regulated environments.
- Knowledge of Master Data Management (MDM), metadata management and data governance frameworks.
- Experience with orchestration tools such as Azure Data Factory or Airflow.
- Azure and/or Databricks certifications.
What's on Offer
- £650 per day Inside IR35.
- Initial 6-month contract with strong potential for extension.
- Opportunity to work on a high-profile Databricks and cloud transformation programme.
- Hybrid working model with 3 days per week onsite in London.
- Collaborative environment working alongside senior technology and business stakeholders.