Data Engineer

About 10Pearls

We are 10Pearls, an award-winning digital development company, that helps businesses with product design, development, and technology acceleration. We have a culture of innovation, uniquely designed to help companies transform, digitalize and scale by levering digital technology.

Wa are looking for a Senior Data Engineer to join our team.

Responsibilities

• Data Ingestion: Work on ingesting a variety of healthcare data (provider, payer, claims, etc.) from multiple sources and vendors, ensuring it is converted into Cohere’s common data model.
• Pipeline Optimization: Consolidate and streamline redundant data pipelines to improve scalability and performance.
• Metadata-Driven Design: Design and implement a metadata-driven approach for data ingestion, accounting for schema drift and providing robust error handling strategies.
• Collaboration: Collaborate with cross-functional teams to understand data requirements and identify areas of improvement in existing processes and pipelines.
• Scalability and Efficiency: Work towards making the data ingestion system more scalable and adaptable to changing data sources and formats.
• Technology Integration: Leverage technologies like dbt and PySpark to enhance data processing capabilities and improve overall data workflows.

Requirements:

• Experience with dbt (Data Build Tool) or PySpark. Experience with dbt is preferred, but PySpark experience will also be valuable.
• Proven ability to design and implement metadata-driven approaches for data ingestion.
• Strong knowledge of handling schema drift and implementing error handling within data pipelines.
• Familiarity with healthcare data (provider, payer, claims) is a plus.
• Ability to work with diverse data formats from multiple vendors and transform them into a unified data model.
• Experience with designing scalable and efficient data systems.

What we offer:
• Work from home
• Flexible Schedules
• Paid PTO
• Amazing People oriented organizational culture
• Challenging projects using the latest technologies with clients from the US and Canada
• Technology and Soft Skills Internal Training
•  Online Courses from AdworldPrime