Skill Profile

• PySpark - Advanced (P3)
• AWS - Advanced (P3)
• SAS - Foundational (P1)

Key Responsibilities Technical Delivery

• Design, develop, and maintain complex PySpark solutions for ETL/ELT and data mart workloads.
• Convert and refactor Legacy SAS code into optimized PySpark solutions using automated tooling and manual refactoring techniques.
• Build scalable, maintainable, and production-ready data pipelines.
• Modernize Legacy data workflows into cloud-native architectures.
• Ensure data accuracy, quality, integrity, and reliability across transformation processes.

Cloud & Data Engineering (AWS-Focused)

• Develop and deploy data pipelines using AWS services such as EMR, Glue, S3, and Athena.
• Optimize Spark workloads for performance, scalability, partitioning strategy, and cost efficiency.
• Implement CI/CD pipelines and Git-based version control for automated deployment.
• Collaborate with architects, engineers, and business stakeholders to deliver high-quality cloud data solutions.

Core Technical Skills PySpark & Data Engineering

• 5+ years of hands-on PySpark experience (Advanced level).

• Strong ability to write production-grade, maintainable data engineering code.

• Solid understanding of:

  • ETL/ELT design patterns
  • Data modelling concepts
  • Fact and dimension modelling
  • Data marts
  • Slowly Changing Dimensions (SCDs)

Spark Performance & Optimization

• Expertise in Spark execution planning, partitioning strategies, and performance tuning.
• Experience troubleshooting distributed data pipelines at scale.

Python & Engineering Quality

• Strong Python programming skills with emphasis on clean, modular, and maintainable code.

• Experience applying engineering best practices including:

  • Parameterization
  • Configuration management
  • Structured logging
  • Exception handling
  • Modular design principles

SAS & Legacy Analytics (Foundational)

• Working knowledge of Base SAS, Macros, and DI Studio.
• Ability to interpret and analyze Legacy SAS code for migration to PySpark.

Data Engineering & Testing

• Understanding of end-to-end data flows, orchestration frameworks, pipelines, and change data capture (CDC).
• Experience creating ETL test cases, unit tests, and data comparison/validation frameworks.

Engineering Practices

• Proficient in Git workflows, branching strategies, pull requests, and code reviews.
• Ability to document technical decisions, architecture, and data flows.
• Experience with CI/CD tooling for data engineering pipelines.

AWS & Platform Expertise (Advanced)

Strong hands-on experience with:

• Amazon S3
• EMR and AWS Glue
• Glue Workflows
• Amazon Athena
• IAM
• Solid understanding of distributed computing and big data processing in AWS environments.
• Experience deploying and operating large-scale data pipelines in the cloud.

Desirable Experience

• Experience within banking, financial services, or other regulated industries.
• Background in SAS modernization or cloud migration programs.
• Familiarity with DevOps practices and infrastructure-as-code tools such as Terraform or CloudFormation.
• Experience working in Agile or Scrum delivery environments.