Data Architect at Sabesp defining and documenting software architectures, system integration strategies and ensuring security and scalability.
About the role
- Senior Data Engineer leading the design and optimization of scalable data architectures on AWS. Collaborating on complex data pipelines and mentoring junior engineers.
Responsibilities
- Lead the design and development of scalable, high-performance data architectures on AWS, leveraging services such as S3, EMR, Glue, Redshift, Lambda, and Kinesis.
- Architect and manage Data Lakes for handling structured, semi-structured, and unstructured data.
- Design and build complex data pipelines using Apache Spark (Scala & PySpark), Kafka Streams (Java), and cloud-native technologies for batch and real-time data processing.
- Optimize these pipelines for high performance, scalability, and cost-effectiveness.
- Develop and optimize real-time data streaming applications using Kafka Streams in Java.
- Build reliable, low-latency streaming solutions to handle high-throughput data, ensuring smooth data flow from sources to sinks in real time.
- Manage Snowflake for cloud data warehousing, ensuring seamless data integration, optimization of queries, and advanced analytics.
- Implement Apache Iceberg in Data Lakes for managing large-scale datasets with ACID compliance, schema evolution, and versioning.
- Design and maintain highly scalable Data Lakes on AWS using S3, Glue, and Apache Iceberg.
- Work with business stakeholders to create actionable insights using Tableau.
- Build data models and dashboards that drive key business decisions, ensuring that data is easily accessible and interpretable.
Requirements
- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field (or equivalent work experience).
- 5+ years of experience in Data Engineering or a related field, with a proven track record of designing, implementing, and maintaining large-scale distributed data systems.
- Proficiency in Apache Spark (Scala & PySpark) for distributed data processing and real-time analytics.
- Hands-on experience with Kafka Streams using Java for real-time data streaming applications.
- Strong experience in Data Lake architectures on AWS, using services like S3, Glue, EMR, and data management platforms like Apache Iceberg.
- Proficiency in Snowflake for cloud-based data warehousing, data modeling, and query optimization.
- Expertise in SQL for querying relational and NoSQL databases, and experience with database design and optimization.
Benefits
- Lead and mentor junior engineers, fostering a culture of collaboration, continuous learning, and technical excellence.
- Ensure high-quality code delivery, adherence to best practices, and optimal use of resources.
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Analytics Data Engineer supporting IHIE's data roadmap by designing pipelines and optimizing datasets. Requires collaboration with teams to enhance healthcare data analytics in Indiana.
Data Architecture Manager shaping enterprise data architecture for Anglian Water. Leading a team to drive data governance, integration, and AI - readiness across the organisation.
SAP Analyst in charge of SAP Data Migration solutions with a focus on business growth. Engaging clients and advising on leading practices to drive SAP success.
Designing, building, and operating data architecture on AWS for Bring! Labs. Leading data migration efforts and collaborating with product and operations teams.
Senior Data Engineer responsible for building data pipelines and infrastructure for AI integration in European companies. Join our team to shape exciting projects in a collaborative environment.
Data Engineer developing and maintaining data ingestion pipelines for an AI startup in public affairs. Collaborating with engineers to enhance data systems for expanded market reach.
Senior Data Engineer responsible for designing and implementing ETL pipelines. Collaborating with cross - functional teams to ensure data quality and optimizing data workflows with AWS tools.
Data Engineer responsible for developing and maintaining data pipelines at Beghou Consulting. Collaborating with teams to optimize data processes for life sciences companies.
Senior Data Engineer building front - end and back - end infrastructure for enterprise data platform at Beghou Consulting. Supporting client teams and developing analytics tools in life sciences.