Make yourself visible and let companies apply to you.
Roles

PySpark Jobs in London

Overview

Looking for PySpark jobs in London? Discover the latest opportunities in one of the UK's top tech hubs on Haystack. Whether you’re a data engineer, developer, or big data specialist, our curated London PySpark job listings connect you with leading employers seeking your skills. Start your next career move today with Haystack – your gateway to the best PySpark roles in London.
Filters applied
London
PySpark
Search
Salary
Location
Remote preference
Role type
Seniority
Tech stack
Sectors
Contract type
Company size
Visa sponsorship
MLOps Tech Lead
Stackstudio Digital Ltd.
London
Hybrid
Senior
£500/day - £525/day
RECENTLY POSTED
processing-js
aws
mongodb
mysql
tensorflow
git
+13
Job Details Role / Job Title: MLOps Tech Lead Work Location: London, UK Office Requirement (Hybrid): 2 days per week Key Responsibilities (High-Level) Data Pipeline Development: Lead the technical direction of projects and ensure the use of Sainsbury’s best practices to the best quality.Data Integration: Lead and provide expertise on Integrate data from various sources, ensuring data consistency, integrity, and quality across the entire data lifecycle.Infrastructure Management: Provide guidance for the junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). The Role As a Tech Lead , you will play a critical role in designing, building, and maintaining data pipelines and infrastructure that enable the development and deployment of machine learning models and drive engineering excellence. You will collaborate closely with data scientists, and lead ML engineers, and software engineers to ensure data is clean, accessible, and optimised for large-scale processing and analysis. Your Responsibilities Data Pipeline Development: Lead the technical direction of projects and ensure the use of Sainsbury’s best practices to the best quality.Data Integration: Lead and provide expertise on Integrate data from various sources, ensuring data consistency, integrity, and quality across the entire data lifecycle.Infrastructure Management: Provide guidance for the junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop).Data Preparation: Collaborate with data scientists to prepare and transform raw data into formats suitable for machine learning, including feature engineering and data augmentation.Automation: Implement automation tools and frameworks (CI/CD) to streamline the deployment and monitoring of machine learning models in production.Performance Optimisation: Optimise data processing workflows and storage solutions to improve performance and reduce costs.Collaboration: Work closely with cross-functional teams, including data science, engineering, and product management, to deliver data solutions that meet business needs.Mentorship: junior and mid-level data engineers and provide technical guidance on best practices and emerging technologies in data engineering and machine learning and helping to enhance their skills and career growth.Knowledge Sharing and Empowerment: Promote a culture of knowledge sharing within the engineering teams by organising regular technical workshops, brown bag sessions, and code reviews.Innovation and Continuous Improvement: Foster a collaborative and inclusive team environment that encourages continuous learning and improvement. Your Profile Essential Skills / Knowledge / Experience Knowledge of machine learning frameworks (e.g., PySpark, PyTorch) and model deployment tools (e.g., MLflow, TensorFlow Serving).Strong experience with data processing frameworks (e.g., Apache Spark, Flink).Expertise in SQL and NoSQL databases (e.g., MySQL, PostgreSQL, MongoDB, Cassandra).Hands-on experience with cloud platforms (e.g., AWS, GCP, Azure) and their data services (e.g., Snowflake, S3, BigQuery, Redshift).Experience with containerisation and orchestration tools (e.g., Docker, Kubernetes).Familiarity with version control systems (e.g., Git) and CI/CD pipelines. Desirable Skills / Knowledge / Experience Certifications: AWS Certified Big Data Specialty, Google Professional Data Engineer, or equivalent. Soft Skills:o Excellent problem-solving and analytical skills.o Strong communication skills, with the ability to explain complex technical concepts to non-technical stakeholders.o Ability to work independently and in a team-oriented, collaborative environment. Leadership and Communication Strong leadership skills with the ability to inspire and guide team.Lead scrum ceremonies as and when needed (Standup, Planning, and grooming sessions).Excellent verbal and written communication skills, with the ability to articulate complex technical concepts.Creating a safe and inclusive environment where all team members feel that their input is valued and are never dissuaded from speaking up or asking questions. Collaborative Attitude Strong team player with a collaborative approach to working with cross-functional teams within the Media Agency.Open to feedback and willing to provide constructive criticism to others.Be available for the team, responding within a reasonable time frame and if not possible clearly sign positing alternative contacts who can guide.Building a community across Media Agency.Contribute to a positive and inclusive atmosphere within the team. Knowledge Sharing and Empowerment Commitment to fostering a learning culture within the team and ensuring knowledge transfer across all levels.Support and mentor C3s and C4s engineers by providing them opportunities to lead initiatives and contribute to the technical roadmap.TPBN1_UKTJ
AI Technical Architect
Purview Consultancy Services Ltd
London
Hybrid
Senior - Leader
Private salary
RECENTLY POSTED
processing-js
python
pyspark
Job Title: AI Technical Architect Location: London, UK (2 days in a week from office, Hybrid)Job type: 6 Months contract with possible extension Active SC Cleared - Inside IR3515 years of IT experience currently hands on with minimum of 5 years of experience on Azure having technically guided managed and governed the team.Primary skills:
Strong communication skills and experience in managing various stakeholder relationships to gain consensus on complex technical solutions
Experience in architecting designing and implementing solutions on-premises in the cloud and using hybrid models
Handson experience in deploying a variety of generative models
In-depth experience in finetuning and customizing pretrained AI models with good understanding of various patterns and practices in AI data engineering and large data processing.
Handson in Prompt Engineering Azure Open AI Form Recognizer Cognitive Search Vector Databases.
Develop and deliver upskilling sessions to the customer.
Python and PySpark.
This role requires the candidate who is already holding active Security Check (SC) clearance in accordance with UK Government standards.
Secondary Skills:
MLOps and LLMOps
Certifications Must Have:
Microsoft Certified Azure Developer Associate AZ204
Certifications Good to Have:
Microsoft Certified Solutions Architect Expert AZ303 AZ304
AI102 Microsoft Certified Azure AI Engineer Associate
DP100 Microsoft Certified Azure Data Scientist Associate
Databricks Professional Certificate in Large Language Models
Microsoft Certified Azure Solutions Architect Expert
Soft Skills:
Good customer connects Prepare solution presentations
Positive attitude and excellent communication skills to manage customer calls
Excellent problem-solving skills good communication
Senior Data Engineer - (ML and AI Platform)
Datatech
London
Hybrid
Senior
£65,000 - £80,000
RECENTLY POSTED
aws
python
sql
pyspark
snowflake
Senior Data Engineer (ML and AI Platform) Location London with hybrid working Monday to Wednesday in the office Salary 65,000 to 80,000 depending on experience Reference J13026We are partnering with an AI first SaaS business that turns complex first party data into trusted, decision ready insight at scale.You will join a collaborative data and engineering team building a modern, cloud agnostic data and AI platforms.This role is well suited to an experienced data engineer who enjoys working thoughtfully with real world data, contributing to reliable production systems, and developing clear and well-structured Python and SQL.Why join: Supportive and inclusive culture where people are encouraged to contribute and be heard Clear progression with space to develop your skills at a sustainable pace An environment where collaboration, learning, and thoughtful engineering are genuinely valuedWhat you will be doing: Contributing to the design and delivery of cloud-based data and machine learning pipelines Working with Python, PySpark and SQL to build clear and maintainable data transformations Helping shape scalable data models that support analytics, machine learning, and product features Collaborating closely with Product, Engineering, and Data Science teams to deliver meaningful production outcomesWhat we are looking for: Experience using Python for data transformation, ideally alongside PySpark Confidence working with SQL and production data models Experience working with at least one modern cloud data platform such as GCP, AWS, Azure, Snowflake, or Databricks Experience contributing to data pipelines that run reliably in production environments A collaborative mindset with clear and thoughtful communicationRight to work in the UK is required. Sponsorship is not available now or in the future.Apply to learn more and see if this could be the next step for you.If you have a friend or colleague who may be interested, referrals are welcome. For each successful placement, you will be eligible for our general gift or voucher scheme. Datatech is one of the UK’s leading recruitment agencies specialising in analytics and is the host of the critically acclaimed Women in Data event. For more information, visit (url removed)
Senior Manager - Palantir Foundry Decision Intelligence Practice
Staffworx Limited
London
Hybrid
Senior
Private salary
RECENTLY POSTED
python
typescript
java
sql
pyspark
Decision Intelligence - Palantir Foundry, Lead Consultant, Senior ManagerWe are looking for a Senior Manager with deep Palantir Foundry expertise to lead the design and delivery of production-grade data and AI solutions. You will shape end-to-end architectures, lead multidisciplinary teams and work directly with senior client stakeholders to turn complex data, AI and process challenges into scalable Foundry applications.Key responsibilities
Act as lead architect for Foundry, owning solution design from ingestion and pipelines through Ontology, applications and AI use cases.
Translate business problems into Foundry use cases, technical designs and deliverable roadmaps.
Design and oversee data pipelines, Ontology models, security and governance patterns and application workflows in Foundry.
Guide teams of data engineers, software engineers and data scientists to deliver robust, secure and maintainable Foundry solutions.
Integrate Foundry with wider enterprise platforms, cloud environments and downstream analytics tools.
Build trusted relationships with senior stakeholders, shaping new opportunities and ensuring value realisation from the platform.
Skills and experience
Significant hands-on experience delivering Palantir Foundry solutions in complex client environments.
Deep Foundry technical expertise across the full stack: Pipeline Builder, Ontology, Workshop, OSDK, Code Repositories, Actions and AIP or agentic capabilities, able to build production-grade applications not just prototypes.
Strong proficiency in at least one relevant programming language such as Python or PySpark, Java, Typescript or SQL.
Solid understanding of data engineering, data modelling, security and governance in enterprise settings.
Experience with software engineering best practices including Git-based development, testing and CI or CD.
Excellent communication and stakeholder management skills, with the ability to influence and align diverse technical and business audiences.
Proven leadership in building, coaching and motivating technical teams.
Sector experience in Financial Services, Government, Healthcare, Energy or Manufacturing is desirable.
Eligibility for, or current possession of, government security clearance is an advantage.
What you will receiveYou will join a specialist Foundry community, working on high-impact programmes with strong support for ongoing learning and certification. A competitive package typically includes flexible and hybrid working, health and wellbeing benefits, professional development support and paid volunteering or community days.
Senior Data Engineer
Tenth Revolution Group
Multiple locations
Fully remote
Senior
£60,000 - £65,000
RECENTLY POSTED
fabric
python
sql
pyspark
About the Role We are looking for a Senior Data Engineer to join a leading Microsoft partner that is modernising data platforms and delivering innovative analytics solutions for organisations across the UK. You will work closely with clients to understand their business challenges before designing tailored solutions that improve efficiency, drive self‑service reporting and support long‑term scalability. This is a hands‑on role where you will support clients from a variety of different sectors. You will also be able to supplement this hands-on experience with the opportunity to gain Microsoft focus certifications and accreditations. Responsibilities Build and manage data pipelines using Azure Synapse, Data Factory, Databricks or Microsoft Fabric Design, implement and maintain data lakes data warehouses and ETL/ELT processes Develop scalable data models for reporting in Power BI Work closely with stakeholders to understand business needs and advise on solutions that best fit the individual needs of the businessSkills and Experience Hands‑on experience Azure services such as Synapse, Data Factory or Databricks Strong SQL skills Proficiency in Python and/or PySpark Experience with Power BI and data modellingWhat is on offer Salary up to £65,000 Fully remote working from anywhere in the UK Performance‑related bonus scheme Pension scheme and private healthcare optionsThis is just a brief overview of the role. For the full details, simply apply with your CV and we’ll be in touch to discuss it further. Tenth Revolution Group are the go‑to recruiter for Data & AI roles in the UK, offering more opportunities nationwide than any other recruitment agency. We are proud sponsors of SQLBits, Power Platform World Tour and the London Fabric User Group
Tech Lead / Lead Data Engineer - Outside IR35 - SC + NPPV3 Cleared
SR2
London
Hybrid
Senior
£500/day - £550/day
RECENTLY POSTED
aws
terraform
github
python
amazon-s3
sql
+3
Tech Lead / Lead Data Engineer (AWS Data Platform) Rate: £500 - £550 p/d outside IR35 Length: 1st April to end of November (initially) Location: London (hybrid – typically 1 day per week on-site, remaining remote) Security Clearance: SC Clearance essential + NPPV3 Overview We’re looking for a hands-on Tech Lead to lead a small team delivering secure, scalable data solutions within a highly regulated environment. You’ll take technical ownership across an AWS-based data platform using S3, Glue, and Redshift, working closely with delivery leadership, architecture stakeholders, and product teams to deliver incremental value. This role suits someone who can balance technical leadership, hands-on engineering, and stakeholder-facing communication, while maintaining strong standards around security, quality, and operational resilience. Key Responsibilities Lead and mentor a small engineering team across data engineering, analytics engineering, and DevOps. Own the technical design of data ingestion, transformation, storage, and access patterns. Drive engineering standards including code quality, testing, CI/CD, Infrastructure as Code, and security-by-design. Translate high-level requirements into solution increments, technical designs, and well-scoped delivery tickets. Deliver and optimise data modelling approaches (e.g., star/snowflake schemas) and performance tuning practices. Build reliable and cost-effective ETL/ELT pipelines, including orchestration and event-driven patterns where appropriate. Partner with security stakeholders to ensure compliance, including IAM least privilege, encryption, auditability, and secure access controls. Implement and maintain CI/CD pipelines for data workflows and platform components. Ensure strong monitoring and operational discipline using cloud-native tooling and engineering best practice. Communicate technical decisions, trade-offs, risks, and delivery progress to senior stakeholders. Promote a culture of learning, quality, and continuous improvement.Required Skills & Experience Proven experience as a Tech Lead / Lead Data Engineer delivering AWS-based data platforms. Strong hands-on AWS experience, including: Amazon S3 (data lake patterns, partitioning, lifecycle policies, cost optimisation) AWS Glue (Jobs, Crawlers, PySpark, Glue Data Catalog, orchestration) Amazon Redshift (performance tuning, sort/dist keys, Spectrum, WLM) Strong development skills across: Python (including PySpark) SQL (DDL/DML, analytical queries, data performance considerations) Experience with Infrastructure as Code (Terraform or CloudFormation). CI/CD experience using tools such as GitHub Actions, Azure DevOps, CodePipeline, CodeBuild, etc. Strong understanding of security & governance in regulated environments: IAM, KMS encryption, Secrets Manager/SSM, audit logging Delivery capability across Agile (Scrum/Kanban) environments with strong backlog refinement discipline. Confident stakeholder management with the ability to explain technical choices and gain consensus
Data Engineer Manager
Young's Employment Services Ltd
Brent
Hybrid
Senior - Leader
£90,000
RECENTLY POSTED
fabric
aws
kafka
python
java
apache-spark
+4
Hybrid - London with 2/3 days WFH Circ £85,000 - £95,000 + Attractive Bonus & Benefits Hands On Data Engineer Manager required for this exciting newly created position with a prestigious and rapidly expanding business in West London. It would suit someone with official management experience, or potentially a Lead / Senior Engineer looking to take on more managerial responsibility. The Data Engineer Manager will play a pivotal role at the heart of our client’s data & analytics operation. Having implemented a new MS Fabric based Data platform, the need now is to scale up and meet the demand to deliver data driven insights and strategies right across the business globally. There’ll be a hands-on element to the role as you’ll be troubleshooting, reviewing code, steering the team through deployments and acting as the escalation point for data engineering. Our client can offer an excellent career development opportunity and a vibrant, creative and collaborative work environment. This is a hybrid role based in Central / West London with the flexibility to work from home 2 or 3 days per week. Key Responsibilities include; Define and take ownership of the roadmap for the ongoing development and enhancement of the Data Platform. Design, implement, and oversee scalable data pipelines and ETL/ELT processes within MS Fabric, leveraging expertise in Azure Data Factory, Databricks, and other Azure services. Advocate for engineering best practices and ensure long-term sustainability of systems. Integrate principles of data quality, observability, and governance throughout all processes. Participate in recruiting, mentoring, and developing a high-performing data organization. Demonstrate pragmatic leadership by aligning multiple product workstreams to achieve a unified, robust, and trustworthy data platform that supports production services such as dashboards, new product launches, analytics, and data science initiatives. Develop and maintain comprehensive data models, data lakes, and data warehouses (e.g., utilizing Azure Synapse). Collaborate with data analysts, Analytics Engineers, and various stakeholders to fulfil business requirements. Key Experience, Skills and Knowledge: Experience leading data or platform teams in a production environment as a Senior Data Engineer, Tech Lead, Data Engineering Manager etc. Proven success with modern data infrastructure: distributed systems, batch and streaming pipelines Hands-on knowledge of tools such as Apache Spark, Kafka, Databricks, DBT or similar Experience building, defining, and owning data models, data lakes, and data warehouses Programming proficiency in the likes of Python, Pyspark, SQL, Scala or Java. Experience operating in a cloud-native environment such as Azure, AWS, GCP etc ( Fabric experience would be beneficial but is not essential). Excellent stakeholder management and communication skills. A strategic mindset, with a practical approach to delivery and prioritisation. Proven success with modern data infrastructure: distributed systems, batch and streaming pipelines. Experience building, defining, and owning data models, data lakes, and data warehouses. Exposure to data science concepts and techniques is highly desirable. Strong problem-solving skills and attention to detail. Salary is dependent on experience and expected to be in the region of £85,000 - £95,000 + an attractive bonus scheme and benefits package. For further information, please send your CV to Wayne Young at Young’s Employment Services Ltd. YES are operating as both a recruitment Agency and Recruitment Business. TPBN1_UKTJ
Senior Azure Data Engineer
Youngs Employment Services
London
Hybrid
Senior
£70,000 - £80,000
RECENTLY POSTED
processing-js
r
python
azure-databricks
delta-lake
sql
+1
Hybrid - Work From Home and West LondonCirc £70,000 - £80,000 + Range of benefitsA well-known and prestigious business is looking to add a Senior Azure Data Engineer to their data team. This is an exciting opportunity for a Data Engineer that’s not just technical, but also enjoys directly engaging and collaborating with stakeholders from across business functions. Having nearly completed the process of migrating data from their existing on-prem databases to an Azure Cloud based platform, the Senior Data Engineer will play a key role in helping make best use of the data by gathering and agreeing requirements with the business to build data solutions that align accordingly. Working with diverse data sets from multiple systems and overseeing their integration and optimisation will require raw development, management and optimisation of data pipelines using tools in the Azure Cloud. Our client has expanded rapidly in recent years, they’re an iconic business with a special work environment that’s manifested a strong and positive culture amongst the whole workforce. This is a hybrid role where the postholder can work from home 2 or 3 days per week, the other days will be based onsite in West London just a few minutes walk from a Central Line tube station.The key responsibilities for the post include;* Develop, construct, test and maintain data architectures within large scale data processing systems.* Develop and manage data pipelines using Azure Data Factory, Delta Lake and Spark.* Utilise Azure Cloud architecture knowledge to design and implement scalable data solutions.* Utilise Spark, SQL, Python, R, and other data frameworks to manipulate data and gain a thorough understanding of the dataset’s characteristics.* Interact with API systems to query and retrieve data for analysis.* Collaborate with business users / stakeholders to gather and agree requirements.To be considered for the post you’ll need at least 5 years experience ideally with 1 or 2 years at a senior / lead level. You’ll need to be goal driven and able to take ownership of work tasks without the need for constant supervision. You’ll be engaging with multiple business areas so the ability to communicate effectively to understand requirements and build trusted relationships is a must. It’s likely you’ll have most, if not all the following:* Experience as a Senior Data Engineer or similar* Strong knowledge of Azure Cloud architecture and Azure Databricks, DevOps and CI/CD.* Experience with PySpark, Python, SQL and other data engineering development tools.* Experience with metadata driven pipelines and SQL serverless data warehouses.* Knowledge of querying API systems.* Experience building and optimising ETL pipelines using Databricks.* Strong problem-solving skills and attention to detail.* Understanding of data governance and data quality principles.* A degree in computer science, engineering, or equivalent experience.Salary will be dependent on experience and likely to be in the region of £70,000 - £80,000 although client may consider higher for outstanding candidate. Our client can also provide a vibrant, rewarding, and diverse work environment that supports career development.Candidates must be authorised to work in the UK and not require sponsoring either now or in the future. For further information, please send your CV to Wayne Young at Young’s Employment Services Ltd. Young’s Employment Services acts in the capacity of both an Employment Agent and Employment Business
MLOps Tech Lead
Stackstudio Digital Ltd.
London
Hybrid
Senior
£500/day - £525/day
processing-js
aws
mongodb
mysql
tensorflow
git
+13
Job DetailsRole / Job Title:MLOps Tech LeadWork Location:London, UKOffice Requirement (Hybrid):2 days per weekKey Responsibilities (High-Level) Data Pipeline Development: Lead the technical direction of projects and ensure the use of Sainsbury’s best practices to the best quality. Data Integration: Lead and provide expertise on Integrate data from various sources, ensuring data consistency, integrity, and quality across the entire data lifecycle. Infrastructure Management: Provide guidance for the junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop).The RoleAs a Tech Lead, you will play a critical role in designing, building, and maintaining data pipelines and infrastructure that enable the development and deployment of machine learning models and drive engineering excellence. You will collaborate closely with data scientists, and lead ML engineers, and software engineers to ensure data is clean, accessible, and optimised for large-scale processing and analysis.Your Responsibilities Data Pipeline Development: Lead the technical direction of projects and ensure the use of Sainsbury’s best practices to the best quality. Data Integration: Lead and provide expertise on Integrate data from various sources, ensuring data consistency, integrity, and quality across the entire data lifecycle. Infrastructure Management: Provide guidance for the junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). Data Preparation: Collaborate with data scientists to prepare and transform raw data into formats suitable for machine learning, including feature engineering and data augmentation. Automation: Implement automation tools and frameworks (CI/CD) to streamline the deployment and monitoring of machine learning models in production. Performance Optimisation: Optimise data processing workflows and storage solutions to improve performance and reduce costs. Collaboration: Work closely with cross-functional teams, including data science, engineering, and product management, to deliver data solutions that meet business needs. Mentorship: junior and mid-level data engineers and provide technical guidance on best practices and emerging technologies in data engineering and machine learning and helping to enhance their skills and career growth. Knowledge Sharing and Empowerment: Promote a culture of knowledge sharing within the engineering teams by organising regular technical workshops, brown bag sessions, and code reviews. Innovation and Continuous Improvement: Foster a collaborative and inclusive team environment that encourages continuous learning and improvement.Your ProfileEssential Skills / Knowledge / Experience Knowledge of machine learning frameworks (e.g., PySpark, PyTorch) and model deployment tools (e.g., MLflow, TensorFlow Serving). Strong experience with data processing frameworks (e.g., Apache Spark, Flink). Expertise in SQL and NoSQL databases (e.g., MySQL, PostgreSQL, MongoDB, Cassandra). Hands-on experience with cloud platforms (e.g., AWS, GCP, Azure) and their data services (e.g., Snowflake, S3, BigQuery, Redshift). Experience with containerisation and orchestration tools (e.g., Docker, Kubernetes). Familiarity with version control systems (e.g., Git) and CI/CD pipelines.Desirable Skills / Knowledge / Experience Certifications: AWS Certified Big Data Specialty, Google Professional Data Engineer, or equivalent.Soft Skills: o Excellent problem-solving and analytical skills. o Strong communication skills, with the ability to explain complex technical concepts to non-technical stakeholders. o Ability to work independently and in a team-oriented, collaborative environment.Leadership and Communication Strong leadership skills with the ability to inspire and guide team. Lead scrum ceremonies as and when needed (Standup, Planning, and grooming sessions). Excellent verbal and written communication skills, with the ability to articulate complex technical concepts. Creating a safe and inclusive environment where all team members feel that their input is valued and are never dissuaded from speaking up or asking questions.Collaborative Attitude Strong team player with a collaborative approach to working with cross-functional teams within the Media Agency. Open to feedback and willing to provide constructive criticism to others. Be available for the team, responding within a reasonable time frame and if not possible clearly sign positing alternative contacts who can guide. Building a community across Media Agency. Contribute to a positive and inclusive atmosphere within the team.Knowledge Sharing and Empowerment Commitment to fostering a learning culture within the team and ensuring knowledge transfer across all levels. Support and mentor C3s and C4s engineers by providing them opportunities to lead initiatives and contribute to the technical roadmap.
Python Data Engineer - Hedgefund
Huxley Associates
London
Hybrid
Mid - Senior
£123,256 - £147,908
python
aws
kubernetes
docker
pandas
sql
+1
Python Data Engineer - Multi-Strategy Hedge FundLocation: London Hybrid: 2 days per week on-site Type: Full-timeAbout the RoleA leading multi-strategy hedge fund is seeking a highly skilled Python Data Engineer to join its technology and data team. This is a hands-on role focused on building and optimising data infrastructure that powers quantitative research, trading strategies, and risk management.Key Responsibilities
Develop and maintain scalable Python-based ETL pipelines for ingesting and transforming market data from multiple sources.
Design and manage cloud-based data lake solutions (AWS, Databricks) for large volumes of structured and unstructured data.
Implement rigorous data quality, validation, and cleansing routines to ensure accuracy of financial time-series data.
Optimize workflows for low latency and high throughput, critical for trading and research.
Collaborate with portfolio managers, quantitative researchers, and traders to deliver tailored data solutions for modeling and strategy development.
Contribute to the design and implementation of the firm’s security master database.
Analyse datasets to extract actionable insights for trading and risk management.
Document system architecture, data flows, and technical processes for transparency and reproducibility.
Requirements
Strong proficiency in Python (pandas, NumPy, PySpark) and ETL development.
Hands-on experience with AWS services (S3, Glue, Lambda) and Databricks.
Solid understanding of financial market data, particularly time-series.
Knowledge of data quality frameworks and performance optimisation techniques.
Degree in Computer Science, Engineering, or related field.
Preferred Skills
SQL and relational database design experience.
Exposure to quantitative finance or trading environments.
Familiarity with containerisation and orchestration (Docker, Kubernetes).
What We Offer
Competitive compensation and performance-based bonus.
Hybrid working model: 2 days per week on-site in London.
Opportunity to work on mission-critical data systems for a global hedge fund.
Collaborative, high-performance culture with direct exposure to front-office teams
To Avoid Disappointment, Apply Now!To find out more about Huxley, please visit (url removed)Huxley, a trading division of SThree Partnership LLP is acting as an Employment Business in relation to this vacancy Registered office 8 Bishopsgate, London, EC2N 4BQ, United Kingdom Partnership Number OC(phone number removed) England and Wales
Senior Data Engineer, SQL, RDBMS, AWS, Python, Mainly Remote
Carrington Recruitment Solutions
London
Fully remote
Senior
£85,000 - £95,000
aws
python
sql
celery
rabbitmq
pyspark
Senior Data Engineer, SQL, RDBMS, Python, Celery, RabbitMQ, AWS, Part Central London, Mainly Remote Senior Data Engineer (SQL, RDBMS, Python, AWS) required to work for a fast growing and exciting business based in Central London. However, this role is mainly remote. We need an experienced Data Developer who is a good people person, working with client facing teams outside of Technology, and also mentoring more junior members of the team across Europe. As the company is fast growing, there will be an opportunity to move upwards at certain points throughout your journey. Read on for more details… Responsibilities * Collaborate with product managers and business stakeholders to understand complex business requirements to translate business needs into well-designed and maintainable solutions * Ensure data quality and reliability by implementing robust data quality checks, monitoring, and alerting to ensure the accuracy and timeliness of all data pipelines * Create data governance policies and develop data models and schemas optimized for analytical workloads * Influence the direction for key infrastructure and framework choices for data pipelining and data management * Manage complex initiatives by setting project priorities, deadlines, and deliverables * Collaborate effectively with distributed team members across multiple time zones, including offshore development teams Skills required: * Proven track record building scalable data pipelines (batch and streaming) in production * Expert Python, PySpark, Celery and RabbitMQ skills; deep experience with AWS data stack (Glue, OpenSearch, RDS) * Expert skills within SQL with experience in both transactional RDBMS systems and distributed systems * Hands-on with Lakehouse technologies (Apache Iceberg, S3 Tables, StarRocks) * Strong grasp of data governance, schema design, and quality frameworks * Comfortable leading infrastructure decisions and collaborating across distributed teams This is a fantastic opportunity and salary is dependent upon experience. Apply now for more details
Page 1 of 1

Frequently asked questions

What types of PySpark jobs are available in London?
London offers a diverse range of PySpark jobs including roles such as Data Engineer, Big Data Developer, Data Scientist, and Analytics Engineer working with large-scale data processing using Apache Spark and PySpark.
Do I need specific experience to apply for PySpark jobs in London?
Most PySpark jobs in London require experience with Python programming and Apache Spark. Familiarity with distributed computing, data processing pipelines, and cloud platforms like AWS or Azure is often preferred.
Are remote or hybrid PySpark job opportunities available in London?
Yes, many employers in London offer remote or hybrid working arrangements for PySpark roles, especially following recent trends towards flexible work setups in the tech industry.
What is the typical salary range for PySpark jobs in London?
Salaries for PySpark roles in London vary depending on experience and role complexity but generally range from £50,000 to £90,000 per year, with senior positions offering higher compensation.
How can I improve my chances of getting a PySpark job in London?
Strengthen your Python and Apache Spark skills, work on real-world big data projects, gain experience with cloud services, and keep your CV updated. Networking and applying via specialist job boards like Haystack can also increase your opportunities.