Senior Data Engineer (Databricks & AWS Migration) - 573433

Senior Data Engineer (Databricks & AWS Migration)

This is a Senior Data Engineer role that involves migrating data pipelines to a new platform using Databricks and AWS. The candidate will analyze and understand existing ETL pipelines, data sources, and transformation logic across current platforms.

Key Responsibilities

• Analyze and understand existing ETL pipelines, data sources, and transformation logic across current platforms
• Drive end-to-end migration of data pipelines to the new platform using Databricks
• Design, build, and maintain scalable data pipelines for batch and near real-time processing in the new platform
• Re-engineer legacy ETL/ELT workflows into optimized, scalable, and modular pipelines using Spark/Databricks
• Integrate data from multiple heterogeneous sources and ensure seamless data flow across systems
• Implement data transformation logic using PySpark/SQL on Databricks and ensure adherence to best practices
• Validate migrated pipelines to ensure data accuracy, reconciliation, and consistency with source systems
• Optimize pipelines for performance, scalability, and cost efficiency within cloud environments
• Work closely with business stakeholders, analysts, and data teams to understand data requirements and use cases
• Support reporting, analytics, and downstream applications by providing reliable and high-quality datasets
• Monitor production pipelines, troubleshoot failures, and proactively resolve data-related issues
• Ensure proper logging, alerting, and observability for all pipelines (using tools like CloudWatch / Databricks monitoring)
• Follow data governance, security, and compliance standards during data handling and migration
• Implement CI/CD pipelines for data workflows and maintain proper documentation of pipelines and processes

Requirements

• Strong proficiency in SQL and relational databases (Oracle, PostgreSQL, etc.)
• Hands-on experience with ETL/ELT pipeline development and migration projects
• Strong programming skills in Python and/or Scala
• Good experience with Apache Spark (preferably PySpark)
• Hands-on experience with Databricks (Delta Lake, notebooks, jobs, workflows)
• Strong knowledge of AWS data engineering services including:
• AWS Glue, Lambda, S3, EventBridge, SQS
• Amazon Redshift, Firehose, CloudWatch
• Strong understanding of data modeling, data warehousing concepts, and data lake architectures
• Experience working with large-scale datasets and distributed processing
• Knowledge of pipeline orchestration and workflow management

Work Arrangements

• Hybrid work arrangement: Face-to-face interactions required (Pan India)
• 6+ years of experience
• Contract type