AWS Data engineer

AWS Data Engineer

We are seeking a talented and experienced AWS Data Engineer to join our team. The ideal candidate will develop and manage scalable ETL/ELT pipelines using Apache Spark on AWS Glue/Databricks. You will collaborate closely with business analysts and stakeholders to understand data requirements and ensure data quality, integrity, and consistency across all systems and reports.

Key Responsibilities

• Develop and manage scalable ETL/ELT pipelines using Apache Spark on AWS Glue/Databricks.
• Collaborate with business analysts and stakeholders to understand data requirements.
• Integrate and process data from various structured and unstructured sources (e.g., Azure Data Lake, APIs, SQL databases).
• Implement data governance best practices including lineage, versioning, and metadata management.
• Participate in code reviews, documentation, and adherence to development standards.
• Ensure data quality, integrity, and consistency across all systems and reports.

Requirements

• Strong experience with Databricks (or Apache Spark), including notebooks, Delta Lake, and DBFS.
• Strong experience with AWS Glue including AWS services like S3, IAM, RDS, etc.
• Proficiency in SQL, PySpark, and data wrangling techniques.
• Hands-on experience with CI/CD pipelines using Pulumi/Terraform.
• Strong experience in Power BI.
• Education: BA, BCS, B.COM, MBA, BE, B.TECH, BSC, M.COM, MCA, MCS.

Work Arrangements

• Type: Full-Time
• Location: Bengaluru, Karnataka, India