Platform Engineer for Pfizer’s Data and AI Platforms team, developing Azure solutions and pro - code AI agent applications. Leading engineering and operations for a scalable enterprise - grade platform.
About the role
- Data Platform Engineer developing data ingestion pipelines for a high-volume data processing system at Mercari. Collaborating with teams to ensure successful data integration.
Responsibilities
- Design, develop, and maintain data ingestion pipelines for a high-volume data processing system that collects data from mobile and web applications (clients).
- Develop and maintain streaming data pipelines to ingest raw data and write it to a data warehouse and lake house.
- Implement batch data transformation pipelines.
- Write SQL queries to extract, transform, and load data from various sources.
- Collaborate with data scientists, data analysts, and software engineers to understand data requirements, manage data schemas, and ensure successful data integration.
- Manage and maintain the CI/CD release pipeline.
- Utilize Docker, YAML, Bash scripting, Terraform, and other technologies to automate infrastructure provisioning and deployments.
- Monitor and troubleshoot pipeline issues to ensure smooth data flow and data quality.
- Write clean, maintainable, and well-documented code.
- Develop and execute on the long-term goals and roadmap of the data platform.
- Develop and maintain a very high RPS REST service to receive user events from clients.
- Develop and maintain logging SDK for the server side system.
Requirements
- Resonates with the mission and values of the Mercari Group and its individual companies
- Experience with streaming data processing frameworks like Apache Beam or Spark or Flink.
- Experience with Data Warehouse technologies like Google BigQuery, Amazon Redshift, Hive/Hadoop or Snowflake.
- Experience designing, developing, and operating large-scale services and/or distributed systems or data pipelines using a variety of programming languages including Go, Python, Java, Scala.
- Experience with building APIs and using data serialization formats (e.g., Protobuf, Avro, Parquet).
- Experience in writing design documents or technical proposals and reaching agreements with stakeholders.
- Familiarity with monitoring and alerting tools.
- Experience with Google Cloud Platform (Dataflow, Pubsub, Kubernetes Engine, Compute Engine)
- Experience with Confluent Cloud or Apache Kafka.
- Experience with Workflow engines like Argo Workflow or Apache Airflow.
- Experience publishing and contributing to OSS.
- English: Independent (CEFR - B2)
- Japanese: Independent (CEFR - B2) preferred
Job title
Job type
Experience level
Salary
Not specified
Degree requirement
Tech skills
Location requirements
Senior Embedded Platform Engineer developing low - level embedded software for Ford's Electric Vehicles and the future of transportation. Collaborating with agile teams to ensure functionality and efficiency.
Senior IT Engineer enhancing cloud platform and infrastructure reliability at Xcel Energy. Collaborating with teams to influence platform strategy and deliver high - impact capabilities.
Platform Engineer developing Kubernetes solutions supporting multi - tenant platforms at Bundesdruckerei in Berlin. Collaborating on innovative digital solutions for identity and data protection.
Lead Platform Engineer at PGIM Private Capital focusing on cloud modernization. Collaborate with cross - functional teams to develop cloud - based applications in a hybrid work environment.
AI Platform Engineer designing and deploying AI and ML platforms at Utica National Insurance Group. Collaborating with internal teams to implement AI solutions and establish observability and telemetry.
Head of Platform Engineering at Flutter leading core infrastructure services and cloud platforms. Focused on modernizing systems and improving reliability for critical operations.
Join Bupa as a Senior Platform Engineer to design and build a scalable, secure data platform. Collaborate and mentor in a healthcare and insurance setting with innovative technology.
Senior Consultant delivering solutions for client challenges using Red Hat technologies. Promote and implement best practices while collaborating with diverse teams.
Power Platform Engineer building secure, scalable business applications on Microsoft Platform for legacy modernization. Collaborating with teams to implement optimal app patterns and enhance user experience.