Data Engineering Team Lead - Databricks

We are working with a leading global Financial Services business to hire an experienced Data Engineering Team Lead to head up a team of engineers working across a large-scale enterprise data platform. This is a senior hands-on leadership role requiring both deep technical expertise and the ability to drive delivery, mentor talent, and set standards across a distributed engineering function.

Responsibilities:

• Lead, mentor, and develop a team of data engineers across multiple locations, driving code reviews, design reviews, and a culture of knowledge-sharing
• Own and drive Agile/Scrum delivery processes, ensuring the team operates effectively against roadmap priorities
• Design and develop scalable data solutions on a Lakehouse architecture platform, supporting enterprise-wide data processing and analytics
• Build, optimise, and maintain ETL/ELT pipelines and structured streaming workflows for both batch and real-time data ingestion
• Configure and tune clusters and Spark jobs to deliver consistent performance at scale
• Utilise Delta Live Tables and Unity Catalog to manage data ingestion, transformation, and access governance
• Apply IAM best practices and uphold compliance with data security and governance standards
• Support infrastructure provisioning and resource management using Terraform
• Implement monitoring frameworks covering pipeline performance, data quality, and operational health
• Contribute to technical documentation and promote continuous improvement across the engineering practice

Requirements

• 8+ years in data engineering, with at least 3 years hands-on experience with Databricks
• Proven experience leading and managing a team of data engineers
• Strong Python and Spark programming skills
• Solid AWS experience across core services including S3, Glue, and Lambda
• Deep understanding of data modelling, SQL, and ETL/ELT design patterns
• Experience with Delta Lake, Lakehouse architecture, and Git-based version control
• Demonstrable use of AI tools within a professional development workflow
• Strong leadership, communication, and stakeholder management skills

Desirable:

• Financial services or fund administration background
• Exposure to AI/ML implementation patterns and real-time data processing frameworks
• Multi-cloud experience beyond AWS
• API development or data governance framework experience
• Track record of developing junior and mid-level engineers in a fast-paced environment

Benefits

Salary: to be discussed, depending on experience

Length: Permanent contract