Make yourself visible and let companies apply to you.
Roles
Apache Spark Jobs
Overview
Discover top Apache Spark jobs on Haystack, your go-to IT job board for data engineering and big data roles. Whether you're a Spark developer, architect, or data scientist, explore the latest opportunities to advance your career in distributed data processing and real-time analytics. Find your perfect Apache Spark job today and join leading tech companies driving innovation with big data.
Data Migration Specialist (Defence Sector)
TECHNOLOGY AND RISK RECRUITMENT LTD
London
In office
Mid - Senior
Private salary
RECENTLY POSTED

Role: Data Migration Specialist - Defence Sector Location: London (On-site) Start Date: ASAP Contract: Freelance Duration: 12 Months +
A leading organisation within the defence sector is seeking a Data Migration & Data Synchronisation Specialist to support a large-scale data transformation initiative. This role requires a contractor experienced in designing and delivering secure data migration frameworks across complex enterprise environments, integrating both on-premise and cloud platforms.

Candidates must be freelance contractors and have Security clearance or eligible for Security clearance due to the nature of the programme
The successful consultant will play a key role in building scalable architectures and automated data pipelines to enable accurate, secure, and high-performance data flows across multiple systems.

Key Responsibilities

  • Design and implement data migration and synchronisation architectures across multiple platforms
  • Develop scalable and secure data transfer frameworks across databases, data warehouses, and data lakes
  • Build and maintain automated data pipelines to synchronise internal and external data sources
  • Ensure performance, reliability, and security of data flows across enterprise systems
  • Implement data encryption and compliance standards aligned with defence sector requirements
  • Work with modern data processing technologies and cloud data platforms

Required Technical Expertise

  • Data Migration & Data Synchronisation Architecture
  • SQL / Python / Scala
  • Data Engineering & Data Pipelines
  • Relational & NoSQL Databases
  • Apache Spark
  • Databricks
  • Data Warehousing / Data Lakes
  • Cloud & On-Prem Data Integration

Additional Requirements

  • Freelance contractor only
  • Must be eligible for security clearance
  • Availability to work on-site in LondonIf you are an experienced Data Migration Specialist looking for a long-term freelance engagement within the defence sector, we would be keen to speak with you.
SparkScala Developer
Infoplus Technologies UK Ltd
London
Hybrid
Senior
£450/day
RECENTLY POSTED

Role Details:Position: SparkScala DeveloperLocation: London, UK (Hybrid)Work Mode: Hybrid (3 Days a week from Office)Duration: Initial contract will be for 12 months, however long-term projectRate: Inside IR35- Up to GBP 450/day
?? Visa Note: Open to British Citizenship, or ILR (Indefinite Leave to Remain), or UK Settlement Visa. We do not provide visa sponsorship.
Key Requirements:Must have skills:-Spark & Scala
Nice to have skills:-Spark Streaming, Hadoop,-Hive,-SQL,-Sqoop,- Impala
Detailed Job Description: At least 8+ years of experience and strong knowledge in Scala programming language. Able to write clean, maintainable and efficient Scala code following best practices. Good knowledge on the fundamental Data Structures and their usage At least 8+ years of experience in designing and developing large scale, distributed data processing pipelines using Apache Spark and related technologies. Having expertise in Spark Core, Spark SQL and Spark Streaming. Experience with Hadoop, HDFS, Hive and other BigData technologies. Familiarity with Data warehousing and ETL concepts and techniques Having expertise in Database concepts and SQL/NoSQL operations. UNIX shell scripting will be an added advantage in scheduling/running application jobs. At least 8 years of experience in Project development life cycle activities and maintenance/support projects. Work in an Agile environment and participation in scrum daily standups, sprint planning reviews and retrospectives. Understand project requirements and translate them into technical solutions which meets the project quality standards Ability to work in team in diverse/multiple stakeholder environment and collaborate with upstream/downstream functional teams to identify, troubleshoot and resolve data issues. Strong problem solving and Good Analytical skills. Excellent verbal and written communication skills. Experience and desire to work in a Global delivery environment. Stay up to date with new technologies and industry trends in Development.

Java Software Engineer II
Computer Futures
Manchester
Hybrid
Junior - Mid
£62/hour
RECENTLY POSTED
+6

Software Engineer II
Hybrid: Manchester (1 day a week)
Inside IR35: £500
Duration: 6 months
Tech Stack: Java / Spring Boot, SQL Server, AWS, Kubernetes, CI/CD. Node.js/Python and SSIS experience is a bonus.

A technology-driven team is seeking an experienced Software Engineer II to support the design, build and operation of backend services within a financial data environment. This role is ideal for someone who values strong engineering practices, autonomy, and cross-functional collaboration.

You will help develop and operate backend systems that deliver accurate, timely financial data to critical enterprise processes. The role includes hands-on development, production support, and ensuring compliance with key regulatory frameworks such as SOX and GDPR/PII.

  • Java / Spring Boot
  • Python & Node.js
  • Microsoft SQL Server, SQL optimisation
  • SSIS (bonus)
  • AWS
  • Kubernetes, Docker
  • CI/CD tooling
  • Infrastructure as Code (TypeScript, AWS CDK)
  • Exposure to Apache Spark is a plus

If this sounds like you, please apply.

Please click here to find out more about our Key Information Documents. Please note that the documents provided contain generic information. If we are successful in finding you an assignment, you will receive a Key Information Document which will be specific to the vendor set-up you have chosen and your placement.

To find out more about Computer Futures please visit

Data Engineer - Highly competitive salary
Anson McCade
Bristol
In office
Mid - Senior
Private salary
RECENTLY POSTED

Data Engineer- Highly competitive salary About the Role: Were partnering with a leading technology consultancy that helps organisations harness the power of data to modernise platforms and drive business outcomes. As a Data Engineer, youll be at the forefront of designing and delivering cloud-native solutions on Google Cloud, turning complex datasets into actionable insights. In this role, youll work on diverse projects, from batch and streaming pipelines to data warehouses, data lakes, and AI-powered analytics platforms. This is a hands-on role where your expertise will guide delivery, shape best practices, and mentor other team members. Key Responsibilities : Lead the design, development, and deployment of scalable data pipelines using BigQuery, Dataflow, Dataproc, and Pub/Sub Automate ETL/ELT workflows and orchestrate pipelines with tools such as Cloud Composer Contribute to architecture and end-to-end solution design for complex data platforms Set engineering standards and ensure high-quality code, deployment, and documentation practices Collaborate with clients and internal teams, translating business requirements into practical solutions Mentor and coach junior engineers to grow their skills and adopt best practices What They're Looking For: They're looking for a Data Engineer who can take ownership of complex data solutions while remaining hands-on. You should have: Proven experience building production-ready solutions on Google Cloud Expertise with batch and streaming frameworks like Apache Spark or Beam Strong understanding of data storage, pipeline patterns, and event-driven architectures Experience with CI/CD, version control, automated testing, and Agile delivery Ability to communicate clearly to both technical and non-technical stakeholders Mentoring or coaching experience Bonus skills: Kafka, enterprise data platform migrations, RDBMS experience (Postgres, MySQL, Oracle, SQL Server), and exposure to ML pipelines. Security Eligibility Candidates must be eligible for UK Security Clearance (SC or DV) if required. Why This Role? This is a chance to work on high-impact, cloud-native projects as a Data Engineer, taking ownership of technical decisions, shaping delivery practices, and developing your career. Youll join a supportive environment where mentoring and learning are highly valued, and your work will directly contribute to the success of complex data programmes. Ok I'm In What's Next? Please apply with your latest CV. TPBN1\_UKTJ

Lead Data Engineer
Fruition Group
Leeds
Hybrid
Senior
£80,000
RECENTLY POSTED
+3

Job Title: Lead Data Engineer
Location: Leeds, 2x per week
Salary: Up to £80,000 per annum

Why Apply?
This is an exciting opportunity to work as a Lead Data Engineer delivering scalable, high quality data solutions for a leading client in the technology sector. This position offers professional growth, challenging projects, and access to cutting edge cloud data technologies.

Lead Data Engineer Responsibilities:

  • Design, develop, and optimise robust, scalable data pipelines and architectures to support Business Intelligence and analytics initiatives.
  • Manage and maintain cloud-based data platforms (AWS, Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions.
  • Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt).
  • Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability.
  • Develop, maintain, and enhance reporting and analytics capabilities through tools such as PowerBI, Tableau, or QuickSight.
  • Apply best practices in data governance, data quality, and performance optimisation.
  • Operate in an agile environment, contributing to technical discussions and problem-solving initiatives.

Lead Data Engineer Requirements:

  • Proven experience in building and managing cloud-based data platforms (AWS Redshift/Glue, Azure Data Factory/Synapse, Google BigQuery/Dataflow).
  • Strong programming skills in Python, SQL, and Java for data engineering tasks.
  • Experience designing reliable, maintainable, and high-performance data pipelines and architectures.
  • Broad understanding of data warehousing, data lakes, and lakehouse architectures.
  • Familiarity with Business Intelligence and data visualisation tools.
  • Excellent analytical thinking, attention to detail, and problem-solving skills.
  • Strong collaboration and communication skills, able to work with both technical and non-technical stakeholders.
  • Comfortable with complexity, ambiguity, and working independently or as part of a team in a fast-paced environment.

We are an equal opportunities employer and welcome applications from all suitably qualified persons regardless of their race, sex, disability, religion/belief, sexual orientation or age.

Senior Data Architect
Bright Purple Resourcing
Glasgow
Remote or hybrid
Senior
£100,000
RECENTLY POSTED
+6

Senior Data Architect Data & AI | Remote (Occasional travel to Glasgow or Reading) Salary £115,000 (Package) Are you a Senior or Data Architect who thrives on designing elegant, scalable cloud data solutions? Do you enjoy helping organisations become truly data-guided through modern Data & AI platforms? This is an exciting opportunity to join a growing consultancy environment where technical excellence and collaboration are key. The Role We are seeking a Senior Architectto join an expanding Data & AI team. You will work closely with architects and consultants to design and deliver high-quality, cloud-based data solutions using Microsoft Azure technologies. You will be accountable for the technical delivery of projects and will engage across the full project lifecycle, from presales through to operational handover. What Youll Be Doing

  • Designing and delivering cloud-based data platforms using Azure services such as Databricks, Synapse Analytics, Microsoft Fabric, Azure SQL Database, Data Lake/Blob Storage, Cosmos DB, Azure Data Factory and Power BI
  • Producing conceptual, logical and physical data models optimised for analytical and reporting use cases
  • Contributing to the definition of data architecture frameworks, standards and principles
  • Mapping data from source to target and defining current and future state architectures based on business requirements
  • Ensuring technical readiness, scalability and quality assurance of delivered solutions
  • Translating complex technical concepts for both technical and non-technical stakeholders
  • Producing clear, high-quality technical documentation

About You You will have extensive experience designing and delivering modern data platforms, with:

  • Strong expertise in data warehouses, data lakes, dimensional modelling, and both batch and streaming data processing
  • A solid understanding of data strategy and management, including data governance, data quality, security and compliance
  • Advanced SQL skills, with hands-on experience using Python and PySpark
  • Broad, practical experience across Azure data services and supporting Azure infrastructure
  • Experience with Infrastructure-as-Code (Terraform or Bicep) and CI/CD pipelines (e.g. Azure DevOps)
  • Excellent communication skills and the ability to engage confidently with senior and executive stakeholders

Desirable Experience

  • Agentic AI, Apache Spark, or enterprise architecture frameworks
  • Enterprise data integration and Microsoft BI technologies
  • Experience with AWS or GCP and additional CI/CD tooling (e.g. Jenkins, BitBucket)
  • Relevant certifications such as Azure Architect Expert and/or TOGAF

Bright Purple is an equal opportunities employer: we are proud to work with clients who share our values of diversity and inclusion in our industry.

Principal GCP Data Engineer
Anson McCade
Multiple locations
Hybrid
Senior
£95,000
RECENTLY POSTED

£Up to £95,000 GBP
Hybrid WORKING
Location: Bristol; Gloucester; Cardiff; Corsham; Cheltenham, Bristol, South West - United Kingdom Type: Permanent

Principal GCP Data Engineer
Join an award-winning innovation and transformation consultancy recognised for its cutting-edge work in data engineering, cloud solutions, and enterprise transformation. This organisation is known for bringing ingenuity to life, helping clients turn complexity into opportunity, and fostering a culture where technical specialists thrive and grow.

An opportunity has arisen for a Principal GCP Data Engineer to join the London-based data and analytics practice. This Principal GCP Data Engineer role offers the chance to lead the design and delivery of end-to-end data solutions on Google Cloud Platform for high-profile clients, shaping data strategy and driving technical excellence across complex programmes.

With a reputation for combining breakthrough technologies with pragmatic delivery, the organisation empowers senior data engineers to influence architecture, mentor teams, and deliver production-ready solutions that create lasting impact.

The Role - Principal GCP Data Engineer
The Principal GCP Data Engineer is a senior technical role responsible for leading data engineering solutions, guiding teams, and acting as a subject matter expert in Google Cloud Platform. As a Principal GCP Data Engineer, you will define end-to-end solution architectures, implement best practices, and lead the development of robust, scalable data pipelines.

This role combines hands-on technical leadership with coaching, mentorship, and client engagement, making it ideal for a Principal GCP Data Engineer who enjoys delivering complex solutions while shaping the capabilities of their team and influencing enterprise-wide data strategy.

What You’ll Be Doing as a Principal GCP Data Engineer
As a Principal GCP Data Engineer, you will:

  • Lead the design, development, and delivery of data processing solutions using GCP tools such as Dataflow, Dataproc, and BigQuery
  • Design automated data pipelines using orchestration tools like Cloud Composer
  • Contribute to architecture discussions and design end-to-end data solutions
  • Own development processes for your team, establishing robust principles and methods across architecture, code quality, and deployments
  • Shape team behaviours around specifications, acceptance criteria, sprint planning, and documentation
  • Define and evolve data engineering standards and practices across the organisation
  • Lead technical discussions with client stakeholders, achieving buy-in for solutions
  • Mentor and coach team members, building technical expertise and capability

Key Responsibilities

  • Develop production-ready data pipelines and processing jobs using batch and streaming frameworks such as Apache Spark and Apache Beam
  • Apply expertise in data storage technologies including relational, columnar, document, NoSQL, data warehouses, and data lakes
  • Implement modern data pipeline patterns, event-driven architectures, ETL/ELT processes, and stream processing solutions
  • Translate business requirements into technical specifications and actionable solution designs
  • Work with metadata management and data governance tools such as Cloud Data Catalog, Collibra, or Dataplex
  • Build data quality alerting and data quarantine solutions to ensure downstream reliability
  • Implement CI/CD pipelines with version control, automated tests, and automated deployments
  • Collaborate in Agile teams, using Scrum or Kanban methodologies

Key Requirements
The successful Principal GCP Data Engineer will bring deep technical expertise, client-facing experience, and leadership skills. You will have:

  • Proven experience delivering production-ready data solutions on Google Cloud Platform
  • Strong knowledge of batch and streaming frameworks, data pipelines, and orchestration tools
  • Expertise in designing and managing structured and unstructured data systems
  • Experience translating business needs into technical solutions
  • Ability to mentor and coach teams and guide technical decision-making
  • Excellent communication skills, with the ability to explain technical concepts to technical and non-technical stakeholders
  • A pragmatic approach to problem solving, combined with a drive for technical excellence

Why Join

  • Take a senior technical leadership role as a Principal GCP Data Engineer within a globally recognised innovation and transformation consultancy
  • Lead the delivery of complex data engineering programmes on Google Cloud Platform
  • Shape the data engineering standards, practices, and architecture across client engagements and internal teams
  • Work in a collaborative, inclusive, and learning-focused culture where technical specialists are empowered to grow and succeed

Reference: AMC/AON/PGCPDataEnginer

#aaon

Java Software Engineer II
Computer Futures
Manchester
Hybrid
Junior - Mid
£62/hour
RECENTLY POSTED
+6

Software Engineer II

Hybrid : Manchester (1 day a week)

Inside IR35: £500

Duration: 6 months

Tech Stack: Java / Spring Boot, SQL Server, AWS, Kubernetes, CI/CD. Node.js/Python and SSIS experience is a bonus.

A technology-driven team is seeking an experienced Software Engineer II to support the design, build and operation of backend services within a financial data environment. This role is ideal for someone who values strong engineering practices, autonomy, and cross-functional collaboration.

You will help develop and operate backend systems that deliver accurate, timely financial data to critical enterprise processes. The role includes hands-on development, production support, and ensuring compliance with key regulatory frameworks such as SOX and GDPR/PII.

Java / Spring Boot

Python & Node.js

Microsoft SQL Server , SQL optimisation

SSIS (bonus)

AWS

Kubernetes , Docker

CI/CD tooling

Infrastructure as Code (TypeScript, AWS CDK)

Exposure to Apache Spark is a plus

If this sounds like you, please apply.

Please click here to find out more about our Key Information Documents. Please note that the documents provided contain generic information. If we are successful in finding you an assignment, you will receive a Key Information Document which will be specific to the vendor set-up you have chosen and your placement.

To find out more about Computer Futures please visit

TPBN1_UKTJ

Java Software Engineer II
Computer Futures
Manchester
Hybrid
Junior - Mid
£62/hour
RECENTLY POSTED
+6

Software Engineer II Hybrid: Manchester (1 day a week) Inside IR35: £500 Duration: 6 months Tech Stack: Java / Spring Boot, SQL Server, AWS, Kubernetes, CI/CD. Node.js/Python and SSIS experience is a bonus. A technology‑driven team is seeking an experienced Software Engineer II to support the design, build and operation of backend services within a financial data environment. This role is ideal for someone who values strong engineering practices, autonomy, and cross‑functional collaboration. You will help develop and operate backend systems that deliver accurate, timely financial data to critical enterprise processes. The role includes hands-on development, production support, and ensuring compliance with key regulatory frameworks such as SOX and GDPR/PII. Java / Spring Boot Python & Node.js Microsoft SQL Server, SQL optimisation SSIS (bonus) AWS Kubernetes, Docker CI/CD tooling Infrastructure as Code (TypeScript, AWS CDK) Exposure to Apache Spark is a plusIf this sounds like you, please apply. Please click to find out more about our Key Information Documents. Please note that the documents provided contain generic information. If we are successful in finding you an assignment, you will receive a Key Information Document which will be specific to the vendor set-up you have chosen and your placement. To find out more about Computer Futures please visit Computer Futures, a trading division of SThree Partnership LLP is acting as an Employment Business in relation to this vacancy | Registered office | 8 Bishopsgate, London, EC2N 4BQ, United Kingdom | Partnership Number | OC(phone number removed) England and Wales

Data Engineer
Youngs Employment Services
London
Hybrid
Junior - Mid
£60,000 - £70,000
RECENTLY POSTED
+3

London + 2 or 3 days work from home

Circ £60,000 - £70,000 + Excellent Benefits Package

A fantastic opportunity is available for a Data Engineer that enjoys working in a fast paced and collaborative team playing work environment. Our client has been expanding at a remarkable pace and have transformed their technical landscape with leading edge solutions. Having implemented a new MS Fabric based Data platform, the need is now to scale up and deliver data driven insights and strategies right across the business globally. The Data Engineer will be joining a close-knit team that is the hub of our client’s global data & analytics operation. Previous experience with MS Fabric would be beneficial but is by no means essential. Interested candidates must have experience in a similar role with MS Azure Data Platforms, Synapse, Databricks or other Cloud platforms such as AWS, GCP, Snowflake etc.

Key Responsibilities will include;

* Design, implement, and optimize end-to-end solutions using Fabric components:

* o Data Factory (pipelines, orchestration)

* o Data Engineering (Lakehouse, notebooks, Apache Spark)

* o Data Warehouse (SQL endpoints, schemas, MPP performance tuning)

* o Real-Time Analytics (KQL databases, event ingestion)

* o Manage and enhance OneLake architecture, delta lake tables, security policies, and data governance within Fabric.

* o Build scalable, reusable data assets and engineering patterns that support analytics, reporting, and machine learning workloads.

* Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective solutions.

* Troubleshoot and resolve data-related issues in a timely manner.

Key Experience, Skills and Knowledge:

* Proven 2 yrs+ experience as a Data Engineer or similar role, with a strong focus on PySpark, SQL, Microsoft Azure Data platforms and Power BI an advantage

* Proficiency in development languages suitable for intermediate-level data engineers, such as:

* Python / PySpark: Widely used for data manipulation, analysis, and scripting.

* SQL: Essential for querying and managing relational databases.

* Understanding of D365 F&O Data Structures is highly desirable

* Strong problem-solving skills and attention to detail.

* Excellent communication and collaboration abilities.

This is a hybrid role based in Central / West London with the flexibility to work from home 2 or 3 days per week. Salary will be dependent on experience and expected to be in the region of £60,000 - £70,000 + an attractive benefits package including bonus scheme.

For further information, please send your CV to Wayne Young at Young’s Employment Services Ltd. YES are operating as both a recruitment Agency and Recruitment Business

Data Engineer
Youngs Employment Services
London
Hybrid
Mid
£60,000 - £70,000
RECENTLY POSTED
+3

London + 2 or 3 days work from home
Circ £60,000 - £70,000 + Excellent Benefits Package

A fantastic opportunity is available for a Data Engineer that enjoys working in a fast paced and collaborative team playing work environment. Our client has been expanding at a remarkable pace and have transformed their technical landscape with leading edge solutions. Having implemented a new MS Fabric based Data platform, the need is now to scale up and deliver data driven insights and strategies right across the business globally. The Data Engineer will be joining a close-knit team that is the hub of our client s global data & analytics operation. Previous experience with MS Fabric would be beneficial but is by no means essential. Interested candidates must have experience in a similar role with MS Azure Data Platforms, Synapse, Databricks or other Cloud platforms such as AWS, GCP, Snowflake etc.

Key Responsibilities will include;

  • Design, implement, and optimize end-to-end solutions using Fabric components:
    • o Data Factory (pipelines, orchestration)
    • o Data Engineering (Lakehouse, notebooks, Apache Spark)
    • o Data Warehouse (SQL endpoints, schemas, MPP performance tuning)
    • o Real-Time Analytics (KQL databases, event ingestion)
    • o Manage and enhance OneLake architecture, delta lake tables, security policies, and data governance within Fabric.
    • o Build scalable, reusable data assets and engineering patterns that support analytics, reporting, and machine learning workloads.
  • Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective solutions.
  • Troubleshoot and resolve data-related issues in a timely manner.

Key Experience, Skills and Knowledge:

  • Proven 2 yrs+ experience as a Data Engineer or similar role, with a strong focus on PySpark, SQL, Microsoft Azure Data platforms and Power BI an advantage
  • Proficiency in development languages suitable for intermediate-level data engineers, such as:
    • Python / PySpark: Widely used for data manipulation, analysis, and scripting.
    • SQL: Essential for querying and managing relational databases.
  • Understanding of D365 F&O Data Structures is highly desirable
  • Strong problem-solving skills and attention to detail.
  • Excellent communication and collaboration abilities.

This is a hybrid role based in Central / West London with the flexibility to work from home 2 or 3 days per week. Salary will be dependent on experience and expected to be in the region of £60,000 - £70,000 + an attractive benefits package including bonus scheme.

For further information, please send your CV to Wayne Young at Young’s Employment Services Ltd. YES are operating as both a recruitment Agency and Recruitment Business

Lead AWS Data Engineer
Opus Recruitment Solutions
Multiple locations
In office
Senior
£550/day - £600/day

Lead AWS Data Engineer | Birmingham | Finance | AWS | Java | Outside IR35 | Contract | 12 Months Opus is partnered with financial services client to deliver a major programme of work. You’ll join an established engineering group, working alongside internal teams to build a new reporting workflow, upgrade an existing pipeline, and lead a full data‑sourcing uplift across multiple reporting workflows. The team will also be responsible for helping upgrade the framework for two critical internal workflows. This role will require you to be on site in either the London or Birmingham office 5 days per week, please only apply if you hit this criteria. Required Experience Strong background in data engineering within distributed data environments Hands-on expertise with AWS, Spark, Glue, and Snowflake Experience building and optimising data pipelines & reporting workflows Ability to work closely with internal engineering and controls teams Experience upgrading or modernising existing workflows and frameworks For Lead-level: prior experience leading engineering teams or workstreamsTech Stack AWS (Glue, S3, Lambda, Step Functions) Apache Spark Snowflake Java if you are interested in this role then please apply here or email me your most recent and up to date CV, along with your availability  to (url removed) Lead AWS Data Engineer | Birmingham | Finance | AWS | Java | Outside IR35 | Contract | 12 Months

Data Platform Engineer
OCC Computer Personnel
London
Hybrid
Mid - Senior
Private salary
+3

Data Platform Engineer – London

(AWS, Apache Spark, AWS Glue, Iceberg, S3, RDS, Redshift, Kafka/MSK, Python, Terraform, Ansible, CI/CD, Jenkins, GitLab, Snowflake, Databricks)

Working with an established FinTech client in London who is looking for a Data Platform Engineer to play a key role in defining, building, and evolving their enterprise Data Lakehouse platform during an exciting period of growth. You’ll work closely with Platform Engineering and Application Engineering teams, taking ownership of the infrastructure, patterns, standards,and tooling used to build and operate data products across the business.

The role focuses on ensuring the data platform is resilient,secure, reliable, and cost-effective within an AWS environment. You’ll be responsible for how the platform is operated, maintained, monitored, and extended, with a strong emphasis on observability, fault prevention, and early fault detection across AWS data services.

Automation is central to the way this team works. You’ll design and maintain Infrastructure as Code and Configuration as Code solutions, supported by CI/CD pipelines, to ensure consistent, repeatable deployments and strong governance. You’ll also enhance data lake integration testing, security measures, monitoring, SLAs, and operational metrics.

Working for a tech driven organisation in a collaborative environment, for an organisation that values engineering that values engineering best practises! This client Is offering this role on hybrid basis, looking to be in the office few times per month.

For more information, please get in touch

Senior Data Engineer
Prospect
London
Hybrid
Senior
Private salary
+1

Prospect is looking for someone who is equally passionate about football and analytics and is excited about the possibilities of the intersection of the two. The ideal candidate would have experience as a problem solver, data engineer, and communicator, preferably with a degree in a quantitative field (such as computer science, engineering, physics, statistics or applied mathematics). You’ll work as part of cross-functional teams to help solve challenges and aid decision makers across the sporting landscape, from elite professional teams, to leagues and broadcasters, applying advanced analytics and modelling techniques. Roles & Responsibilities: - A passion for sport with an understanding of our clients’ sporting disciplines or an eagerness to learn about them. - A strong programming proficiency in Python and SQL querying. Experience with relational database platforms. - Knowledge of cloud technologies. It is an advantage (but not a requirement) to have had experience working with AWS. - Excellent collaboration and communication skills. - 4+ years of experience in big data related software development; experience with data modelling, design patterns and building highly scalable and secured solutions. - Practical knowledge of software engineering concepts and best practices, like testing frameworks, packaging, API design, DevOps, DataOps and MLOps. - The right to work in the United Kingdom.

Page 1 of 1
Frequently asked questions
We feature a variety of Apache Spark jobs including Data Engineer, Big Data Developer, Spark Developer, Data Scientist, and Analytics Engineer positions across different industries.
Most Apache Spark jobs require some level of experience with the technology, but entry-level positions and internships are also available for candidates eager to start their careers.
Yes, our platform lists numerous remote Apache Spark opportunities, allowing you to work from anywhere while leveraging your Spark skills.
Employers often seek skills such as Scala, Python, SQL, Hadoop, Kafka, and cloud platforms like AWS or Azure alongside Apache Spark expertise.
To improve your chances, ensure your resume highlights your Spark projects and relevant skills, apply promptly to new listings, and consider upskilling with certifications or courses in big data technologies.