Hybrid Senior Software Engineer, Data Platform

at Parafin

• Job title: Senior Software Engineer, Data Platform
• Job type: Full Time
• Experience level: Senior
• Salary: $230,000 - $265,000 per year
• Degree requirement: Bachelor's Degree
• Tech skills: Airflow, AWS, Cloud, Distributed Systems, Hadoop, Kafka, PySpark, Python, Spark, SQL, Terraform
• Location requirements: Hybrid, San Francisco, United States

About the role

• Senior Software Engineer leading the development of a Data Platform at a fintech startup. Designing scalable data solutions for small businesses' financial services and supporting product development.

Responsibilities

• Design and build robust, highly scalable data pipelines and lakehouse infrastructure with PySpark, Databricks, and Airflow on AWS.
• Improve the data platform development experience for Engineering, Data Science, and Product by creating intuitive abstractions, self‑service tooling, and clear documentation.
• Own and maintain core data pipelines and models that power internal dashboards, ML models, and customer-facing products.
• Own the Data & ML platform infrastructure using Terraform, including end‑to‑end administration of Databricks workspaces: manage user access, monitor performance, optimize configurations (e.g., clusters, lakehouse settings), and ensure high availability of data pipelines.
• Lead projects to improve data quality, testing, observability, and cost efficiency across existing pipelines and backend systems (e.g., migrating Databricks SQL pipelines to dbt, scaling data ingestion, improving data-lineage tracking, and enhancing monitoring).
• Act as the primary engineering partner for the Data Science team—embedded closely to gather requirements, design scalable solutions, and provide end-to-end support on all engineering aspects of their work.
• Work closely with backend engineers and data scientists to design performant data models and support new product development initiatives.
• Share best practices and mentor other engineers working on data-centric systems.

Requirements

• 4+ years of experience in software engineering with a strong background in data infrastructure, pipelines, and distributed systems.
• Advanced proficiency in Python and SQL.
• Hands-on Spark development experience.
• Expertise with modern cloud data stacks—AWS (S3, RDS), Databricks, and Airflow—and lakehouse architectures.
• Hands‑on experience with foundational data‑infrastructure technologies such as Hadoop, Hive, Kafka (or similar streaming platforms), Delta Lake/Iceberg, and distributed query engines like Trino/Presto.
• Familiarity with ingestion frameworks, developer‑experience tooling, and best practices for data versioning, lineage, partitioning, and clustering.
• Strong problem-solving skills and a proactive attitude toward ownership and platform health.
• Excellent communication and collaboration skills, especially in cross-functional settings.

Benefits

• Equity grant
• Medical, dental & vision insurance
• Work from home flexibility
• Unlimited PTO
• Commuter benefits
• Free lunches
• Paid parental leave
• 401(k)
• Employee assistance program