Platform Engineer leading lifecycle management of MOM and AMHS systems across Kubernetes clusters in semiconductor industry. Collaborating with internal teams to ensure operational reliability in manufacturing.
About the role
- Data Platform Engineer developing data ingestion pipelines for a high-volume data processing system at Mercari. Collaborating with teams to ensure successful data integration.
Responsibilities
- Design, develop, and maintain data ingestion pipelines for a high-volume data processing system that collects data from mobile and web applications (clients).
- Develop and maintain streaming data pipelines to ingest raw data and write it to a data warehouse and lake house.
- Implement batch data transformation pipelines.
- Write SQL queries to extract, transform, and load data from various sources.
- Collaborate with data scientists, data analysts, and software engineers to understand data requirements, manage data schemas, and ensure successful data integration.
- Manage and maintain the CI/CD release pipeline.
- Utilize Docker, YAML, Bash scripting, Terraform, and other technologies to automate infrastructure provisioning and deployments.
- Monitor and troubleshoot pipeline issues to ensure smooth data flow and data quality.
- Write clean, maintainable, and well-documented code.
- Develop and execute on the long-term goals and roadmap of the data platform.
- Develop and maintain a very high RPS REST service to receive user events from clients.
- Develop and maintain logging SDK for the server side system.
Requirements
- Resonates with the mission and values of the Mercari Group and its individual companies
- Experience with streaming data processing frameworks like Apache Beam or Spark or Flink.
- Experience with Data Warehouse technologies like Google BigQuery, Amazon Redshift, Hive/Hadoop or Snowflake.
- Experience designing, developing, and operating large-scale services and/or distributed systems or data pipelines using a variety of programming languages including Go, Python, Java, Scala.
- Experience with building APIs and using data serialization formats (e.g., Protobuf, Avro, Parquet).
- Experience in writing design documents or technical proposals and reaching agreements with stakeholders.
- Familiarity with monitoring and alerting tools.
- Experience with Google Cloud Platform (Dataflow, Pubsub, Kubernetes Engine, Compute Engine)
- Experience with Confluent Cloud or Apache Kafka.
- Experience with Workflow engines like Argo Workflow or Apache Airflow.
- Experience publishing and contributing to OSS.
- English: Independent (CEFR - B2)
- Japanese: Independent (CEFR - B2) preferred
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Own product platform and release - quality systems for AI SaaS startup. Implement analytics, build dashboards, and ensure safe releases while maintaining high quality standards.
Principal Cloud Security Design Engineer defining and engineering cloud security architecture. Leading technical initiatives in Azure and AWS environments for financial services company.
Mid - level Platform Engineer for FAA modernization project at OCH Technologies. Responsible for designing, implementing, and managing secure automated platform environments supporting aviation systems.
Hands - on engineer designing, building, and maintaining core backend systems at MyFunded Futures. Leading technical architecture and mentoring the engineering team in a fintech environment.
Software Engineer developing advanced trading applications for professional derivatives traders at TT. Collaborate with the team to enhance the award - winning trading platform.
Senior Platform Engineer helping design, scale, and harden Pivotal’s AI - driven platform. Collaborating closely with engineering teams to improve reliability, security, and scalability.
Senior technical authority at Smarsh managing large - scale distributed data platforms. Leading architectural design, influencing engineering standards, and mentoring engineers across the organization.
Platform Engineer II at Smarsh managing enterprise workloads and Kubernetes - based hosting solutions with a focus on security and scalability.
AI Platform Engineer building post - RPA automation platforms for enterprises. Collaborating on AI - driven solutions across UIs and APIs with a focus on governance and outcomes.