Senior Engineer responsible for data lake management and processing pipeline design using Python, Spark, and Azure. Ensuring high data quality and operational continuity in a growing organization.
Responsibilities
Maintain, develop, and optimize the global datalake environment, ensuring high data quality, reliability, and overall performance
Design and implement high-performance data processing pipelines and workflows utilizing Python, Spark, and Databricks
Manage scalable and efficient data integration and processing solutions using Azure Data Factory, Databricks, and Event Hub
Establish robust data management processes and execute advanced data querying and quality control with SQL and big data technologies
Automate operational tasks and support infrastructure maintenance by leveraging shell scripting and foundational UNIX knowledge
Support data modeling initiatives for coherent structure design and document all technical processes to ensure operational continuity
Requirements
Strong background and understanding of database data management (general SQL knowledge, data querying, data quality control)
Strong programming knowledge in Python (pandas, numpy)
Advanced level Big Data / data lake framework (Spark, Databricks)
Strong cloud knowledge and handling of Azure Data Factory, Databricks, Event Hub
Basic familiarity with UNIX operating system, especially shell scripting
Basic understanding of network level problems and connectivity requirements
Basic Understanding data modelling principes
Data-centric mindset
Structured, analytical thinking
Can work in a multi-shift operation (Monday to Friday, 07:00 AM - 22:00 PM) + on-call (weeknights, weekend)
Benefits
Possibility to improve yourself in a constantly growing organization
Tech Lead managing Data Engineering for a French digital solutions company. Leading data solutions for e - retail performance with Python and SQL on modern architectures.
Staff Software Engineer enhancing TeamViewer ONE capabilities for small and medium businesses. Collaborating to maintain and improve user experiences with distributed systems and cloud platforms.
Senior Director of Software Engineering leading a team focused on AI - enabled technology initiatives. Manage projects that transform business and technology capabilities in the insurance industry.
Senior Software Engineer developing cross - product features for enterprise customers at Cloudera. Collaborating within a global team and ensuring high - quality metadata management services.
IT Cloud Software Architect designing and scaling cloud - native applications at Nelnet. Leading technical direction and fostering innovation in a hybrid work environment.
Software Engineer Lead developing ETL solutions for PNC's regulatory compliance needs. Leading design and development of data solutions with compliance emphasis.
Senior Software Engineer focusing on backend development at CVS Health. Building software components using a cloud - native platform on Google Cloud Platform.
Software Engineer developing high quality products for OPENLANE in web, iOS, and Android environments. Collaborating in an agile team to build solutions with backend microservices on AWS cloud.
Software Engineer supporting BlueCard claims processing by enhancing applications and modernizing legacy systems. Requires experience in COBOL, C#, and SQL Server with remote work options.