Hybrid Data Engineer, Azure, Databricks, Mid/Senior

Posted 3 days ago

Apply now

About the role

  • Engenheiro(a) de Dados responsável por desenvolvimento de pipelines e otimização de dados em Azure. Focando na qualidade de dados e na arquitetura Medallion.

Responsibilities

  • Pipeline Development: Build and maintain robust data pipelines using PySpark in the Azure Databricks environment, ensuring high availability and continuous data flow.
  • Medallion Architecture: Implement and manage the data lifecycle following a layered structure (Bronze, Silver, Gold), ensuring organization and quality from ingestion through consumption.
  • Ingestion and Integration: Configure ETL/ELT workflows from heterogeneous sources (SQL Server, APIs, streaming) using Azure Data Factory and Azure Data Lake Storage (ADLS).
  • Optimization and Monitoring: Oversee cluster and process performance with a focus on scalability, latency reduction, and cost-efficiency in the cloud.
  • Data Quality: Develop data cleansing, processing and enrichment routines to turn raw data into assets ready for business analysis.
  • Security and Governance: Document technical processes and ensure all pipelines comply with data security and privacy policies.

Requirements

  • Education: Bachelor's degree in Computer Science, Engineering, Data Science or related IT fields.
  • Azure Cloud: Hands-on experience with the Microsoft data ecosystem.
  • Technical Stack: Advanced proficiency in Databricks, PySpark, Python and SQL.
  • Architecture: Solid experience with Medallion Architecture, Data Lake concepts and Delta Lake technology.
  • Modeling: Knowledge of data modeling (Dimensional/Star schema).
  • Visualization: Basic knowledge of Power BI to support the consumption (Gold) layer.

Job title

Data Engineer, Azure, Databricks, Mid/Senior

Job type

Experience level

Senior

Salary

Not specified

Degree requirement

Bachelor's Degree

Location requirements

Report this job

See something inaccurate? Let us know and we'll update the listing.

Report job