Hybrid Data Engineer, Hadoop, Scala, Databricks – Assistant Manager

Posted 2 months ago

Apply now

About the role

  • Data Engineer for State Street collaborating on a scalable infrastructure platform across AWS/Azure. Mentoring engineers and transforming data analytics tooling for various business needs.

Responsibilities

  • Collaborate across a variety of teams to enable our Data Platform needs and design, implement, and maintain a secure and scalable infrastructure platform spanning across AWS/Azure and our Data Center
  • Own and ensure that internal and external SLA’s meet and exceed expectations, System centric KPIs are continuously monitored
  • Work on our data lake, data warehouse, and stream processing systems to create a unified query engine, multi-model databases, analytics extracts and reports, as well as dashboard and visualizations
  • Design and build systems for high availability, high throughput, data consistency, security, and end user privacy, defining our next generation of data analytics tooling
  • Mentor other engineers and promote software engineering best practices across the organization designing systems with monitoring, auditing, reliability, and security at their core
  • Come up with solutions for scaling data systems for various business needs and collaborate in a dynamic and consultative environment

Requirements

  • Minimum 8+ years of Databricks experience using Azure Cloud
  • Minimum 5+ years of hands on expertise in Hadoop and Scala or PySpark
  • Knowledge of at least one of the public cloud platforms AWS, Azure, or GCP is required.
  • Databricks Platform Expertise (Notebooks, Clusters, Jobs, Delta Lake)
  • Programming & Data Processing using (Scala, Python, PySpark, SQL)
  • Apache Spark optimization techniques
  • Azure Services (Data Factory, Blob Storage, Azure Databricks)
  • CI/CD with Azure DevOps
  • Secure Coding principles in Databricks, Data Governance and Compliance standards
  • Certification on Azure/AWS (Good to have)
  • Certification on Hadoop, Scala and Databricks (Good to have)
  • Familiarity with SDLC and Agile methodologies

Benefits

  • Generous medical care, insurance and savings plans
  • Flexible Work Programs
  • Wealth of development programs and educational support

Job title

Data Engineer, Hadoop, Scala, Databricks – Assistant Manager

Job type

Experience level

SeniorLead

Salary

Not specified

Degree requirement

Bachelor's Degree

Location requirements

Report this job

See something inaccurate? Let us know and we'll update the listing.

Report job