Senior M365 & IAM Platform Engineer at Aurobay developing hybrid engines and transmissions for a global market. Responsible for optimizing M365, Active Directory, and Entra ID environments with a strong focus on automation.
About the role
- Data Platform Engineer developing data ingestion pipelines for a high-volume data processing system at Mercari. Collaborating with teams to ensure successful data integration.
Responsibilities
- Design, develop, and maintain data ingestion pipelines for a high-volume data processing system that collects data from mobile and web applications (clients).
- Develop and maintain streaming data pipelines to ingest raw data and write it to a data warehouse and lake house.
- Implement batch data transformation pipelines.
- Write SQL queries to extract, transform, and load data from various sources.
- Collaborate with data scientists, data analysts, and software engineers to understand data requirements, manage data schemas, and ensure successful data integration.
- Manage and maintain the CI/CD release pipeline.
- Utilize Docker, YAML, Bash scripting, Terraform, and other technologies to automate infrastructure provisioning and deployments.
- Monitor and troubleshoot pipeline issues to ensure smooth data flow and data quality.
- Write clean, maintainable, and well-documented code.
- Develop and execute on the long-term goals and roadmap of the data platform.
- Develop and maintain a very high RPS REST service to receive user events from clients.
- Develop and maintain logging SDK for the server side system.
Requirements
- Resonates with the mission and values of the Mercari Group and its individual companies
- Experience with streaming data processing frameworks like Apache Beam or Spark or Flink.
- Experience with Data Warehouse technologies like Google BigQuery, Amazon Redshift, Hive/Hadoop or Snowflake.
- Experience designing, developing, and operating large-scale services and/or distributed systems or data pipelines using a variety of programming languages including Go, Python, Java, Scala.
- Experience with building APIs and using data serialization formats (e.g., Protobuf, Avro, Parquet).
- Experience in writing design documents or technical proposals and reaching agreements with stakeholders.
- Familiarity with monitoring and alerting tools.
- Experience with Google Cloud Platform (Dataflow, Pubsub, Kubernetes Engine, Compute Engine)
- Experience with Confluent Cloud or Apache Kafka.
- Experience with Workflow engines like Argo Workflow or Apache Airflow.
- Experience publishing and contributing to OSS.
- English: Independent (CEFR - B2)
- Japanese: Independent (CEFR - B2) preferred
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
DevOps Engineering Manager leading middleware platform engineering team at QVC. Combining technical leadership with hands - on engineering to evolve the enterprise middleware infrastructure.
Senior IAM & SaaS Engineer designing scalable identity solutions at Fox Corporation. Collaborating with cross - functional teams to strengthen security and improve operational efficiency.
Developer focused on Dynamics 365 and Power Platform solutions for Mitacs' digital transformation. You will work with various teams to deliver reliable and secure solutions.
Senior Platform Engineer helping shape platform's technical direction at fintech company BUX. Evolving internal frameworks and rolling out AI coding practices across engineering organization.
Platform Engineer maintaining high - performance infrastructure for Knauf Group's digital transformation. Involving security protocols, automated processes, and architectural improvements.
Senior Platform Engineer automating runtime infrastructure and championing CI/CD for G+D digital security solutions. Collaborating with cross - functional teams on robust monitoring and operations.
Senior Staff Platform Engineer designing solutions for monitoring and troubleshooting at Equinix, joining a globally distributed team on cutting - edge platform infrastructure.
Platform Engineer designing, building, and operating AI - enabled infrastructure for Comcast. Leading the implementation of cloud solutions ensuring scalability, reliability, and integration with applications and data teams.
Senior AI Platform Engineer designing and operating backend services for payroll platform automation. Involves integrating LLMs into production services and developing cloud infrastructure.