Lead Data Engineer - Pyspark / AWS / Python / SAS - Financial Sector

As a Lead PySpark Engineer, you will design, develop, and fix complex data processing solutions using PySpark on AWS. You will work hands-on with code, modernising legacy data workflows and supporting large-scale SAS-to-PySpark migrations. The role requires strong engineering discipline, deep data understanding, and the ability to deliver production-ready data pipelines in a financial services environment.

Essential Skills

PySpark & Data Engineering

• Minimum 5+ years of hands-on PySpark experience.
• SAS to Pyspark migration experience
• Proven ability to write production-ready PySpark code.
• Strong understanding of data and data warehousing concepts, including: ETL/ELT, Data models, Dimensions and facts, Data marts, SCDs

Spark Performance & Optimisation

• Strong knowledge of Spark execution concepts, including partitioning, optimisation, and performance tuning.
• Experience troubleshooting and improving distributed data processing pipelines.
• Python & Engineering Quality
• Strong Python coding skills with the ability to refactor, optimise, and stabilise existing codebases.
• Experience implementing parameterisation, configuration, logging, exception handling, and modular design.

SAS & Legacy Analytics

• Strong foundation in SAS (Base SAS, SAS Macros, SAS DI Studio).
• Experience understanding, debugging, and modernising legacy SAS code.

Data Engineering & Testing

• Ability to understand end-to-end data flows, integrations, orchestration, and CDC.
• Experience writing and executing data and ETL test cases.
• Ability to build unit tests, comparative testing, and validate data pipelines.

Engineering Practices

• Proficiency in Git-based workflows, branching strategies, pull requests, and code reviews.
• Ability to document code, data flows, and technical decisions clearly.
• Exposure to CI/CD pipelines for data engineering workloads.

AWS & Platform Skills

• Strong understanding of core AWS services, including: S3, EMR / Glue, Workflows, Athena, IAM
• Experience building and operating data pipelines on AWS.
• Big data processing on cloud platforms.

Desirable Skills

• Experience in banking or financial services.
• Experience working on SAS modernisation or cloud migration programmes.
• Familiarity with DevOps practices and tools.
• Experience working in Agile/Scrum delivery environments.

I have three roles available all of which can be worked remotely so dont delay and apply today. I have interview slots ready to be filled

Randstad Technologies is acting as an Employment Business in relation to this vacancy.