Hybrid Databricks Platform Engineer

at EXL

• Job title: Databricks Platform Engineer
• Job type: Full Time
• Experience level: Mid level, Senior
• Salary: Not specified
• Degree requirement: Bachelor's Degree
• Tech skills: AWS, Cloud, PySpark, Python, Terraform, Unity
• Location requirements: Hybrid, Colombia

About the role

• Databricks Platform Engineer managing AWS integration and data pipelines for advanced analytics. Driving innovation and optimization in Databricks ecosystem with a focus on machine learning capabilities.

Responsibilities

• Leads the design, build, and optimization of the Databricks ecosystem, including Fivetran ingestion and dbt transformation pipelines.
• Ensures scalable, reliable compute environments, seamless AWS integration, and automated deployments using Terraform and GitHub Actions.
• Manage Fivetran connectors and dbt projects for end-to-end data ingestion, transformation, and orchestration across Databricks, S3, and Snowflake.
• Automate infrastructure provisioning and configuration using Terraform integrated with GitHub Actions.
• Architect and implement high availability (HA) and disaster recovery (DR) strategies, including cross-region replication and failover.
• Implement job orchestration, cluster policies, autoscaling, and CI/CD workflows for notebooks, models, and releases.
• Establish monitoring and alerting for cost, usage, pipeline health, data freshness, and compliance using tools such as Datadog.
• Collaborate with platform segments and security teams to enforce governance, tagging, rightsizing, SLAs, and audit readiness.
• Develop and maintain Unity Catalog for governance, lineage, and secure data sharing across platforms.
• Build and maintain MLflow integrations for model lifecycle management, experimentation tracking, and deployment.

Requirements

• Graduate in Computer Science, Data Science, or related field.
• 4-6 years of experience in data engineering or related field.
• Deep expertise in Databricks administration, cluster management, job scheduling, Unity Catalog, Fivetran, and dbt (Core/Cloud).
• Proficiency in Terraform, GitHub Actions, and Python/PySpark for automation.
• Strong knowledge of AWS services, including EKS, ECS, IAM, S3, VPC networking, and CloudWatch.
• Experience in pipeline reliability engineering, cost optimization, ML CI/CD, HA/DR design, and observability.

Benefits

• Health insurance
• 401(k) matching
• Flexible work hours
• Paid time off
• Professional development opportunities