Founding Staff Data Engineer building and leading data engineering team for AI - driven art valuation platform. Establishing architecture and standards for data systems and pipelines.
About the role
- Senior Data Engineer (AWS) with expertise in Python and data services. Working on enterprise-scale data processing and analytics initiatives in a hybrid model.
Responsibilities
- Design, develop, and maintain scalable data processing pipelines using Python, PySpark, and Spark SQL
- Build and optimize distributed data processing workflows on AWS platforms.
- Leverage AWS data services such as EMR, Glue, Lambda, and S3 for batch and real-time data processing.
- Design and manage data storage solutions using RDS/MySQL, Redshift , and other AWS-native databases.
- Implement effective data modeling, schema design, and schema evolution strategies.
- Perform performance tuning and optimization of Spark jobs and SQL queries.
- Monitor and troubleshoot data pipelines using AWS CloudWatch and logging frameworks.
- Manage secrets and credentials securely using AWS Secrets Manager.
- Collaborate with data architects, analysts, and stakeholders to translate business requirements into technical solutions.
- Debug complex data issues and provide root cause analysis with long-term fixes.
- Ensure data quality, reliability, and scalability across platforms
Requirements
- 10–13 years of overall experience in Data Engineering
- Strong proficiency in Python and SQL
- Extensive hands-on experience with PySpark and Spark SQL
- Strong experience with AWS data services , including: EMR Glue Lambda S3 RDS / MySQL Redshift CloudWatch Secrets Manager
- Solid understanding of distributed computing concepts
- Strong experience in data modeling, schema handling, and performance tuning
- Excellent debugging, analytical, and problem-solving skills.
- Ability to work effectively in a hybrid and collaborative environment
Benefits
- Immediate joiners only
- Hybrid work mode
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Senior Data Engineer responsible for developing, maintaining ETL processes and integrating data solutions. Collaborating with teams on data quality and cloud migration initiatives.
Data Engineer optimizing data architectures and pipelines at Nexu. Focused on building reliable and efficient data flows while collaborating with cross - functional teams.
Senior Software Engineer designing and maintaining scalable data solutions for restaurant tech industry at SpotOn. Collaborating with cross - functional teams to enhance reporting and analytics platforms.
Data Architect needed to define and evolve data architecture supporting scientific compute at EIT. Collaborate and lead in large - scale research environments for transformative scientific challenges.
Engineering Data Coordinator leading a data engineering team in Azure and Databricks at Deroyque. Focusing on project management, quality assurance, and team development based in Campinas.
Data Migration Specialist managing ongoing Salesforce data quality initiatives for Abby Care. Executing and validating data migrations while ensuring data accuracy.
Data Migration Analyst responsible for supporting data migration and data quality initiatives at Abby Care, tackling family caregiving challenges.
Lead Data Engineer overseeing and managing the Data Engineering team. Developing ETL pipelines and ensuring data integrity within Cloud (Azure) infrastructure.
Data Engineer designing and optimizing data solutions for Qualco Intelligent Finance. Focus on data integrity, consistency, and reusability in analytics deliverables within a hybrid environment.