Welcome to 10Pearls!
We believe in harnessing the power of technology for social good through our core values: Innovate, modernize and accelerate.
About us
We are 10Pearls, an award-winning digital development company, helping business with product design, development, and technology acceleration. We have a culture of innovation, uniquely designed to help businesses transform, digitalize and scale levering digital technology.
The Role
The Senior Data Engineering Focus, will be a key contributor to our data platform. This role involves designing, developing, and maintaining scalable and robust data pipelines, data lakes, and lakehouse architectures. The ideal candidate will possess strong expertise in Python, experience with various data warehousing solutions and big data processing frameworks (such as Snowflake, Databricks, or similar technologies), and a solid understanding of modern software development practices.
Key Responsibilities:
- Data Pipeline Development & Optimization: Design, build, and maintain efficient, scalable, and reliable ETL/ELT data pipelines to ingest, process, and transform large volumes of structured and unstructured data from various sources.
- Data Modeling & Architecture: Design and implement data models for optimal storage and retrieval. Play a key role in the architecture, design, and implementation of our data lake and lakehouse solutions, ensuring scalability, performance, and security.
- Data Warehousing & Big Data Technologies: Develop and manage solutions utilizing modern data warehousing platforms and big data technologies (e.g., Apache Spark, cloud-based data warehouses, distributed processing systems). Experience with technologies like Snowflake or Databricks is highly valuable.
- Full-Stack Development for Data Applications: Develop and maintain APIs for data access and integration with other applications and services.
- Data Quality & Governance: Implement data quality checks, validation processes, and monitoring to ensure data accuracy, consistency, and reliability. Adhere to data governance best practices.
- Performance Tuning & Optimization: Monitor and optimize the performance of data pipelines, queries, and data storage systems.
- Collaboration & Mentorship: Work closely with cross-functional teams including data scientists, analysts, product managers, and other engineers. Mentor junior team members and promote best practices in data engineering and software development.
- Innovation & Emerging Technologies: Stay current with the latest trends and technologies in data engineering, big data, cloud computing, and analytics, and proactively identify opportunities for innovation and improvement.
Required Qualifications:
- Bachelor's or Master's degree in Computer Science, Engineering, or a related technical field.
- 5+ years of professional experience in software engineering, with a significant focus on data engineering.
- Strong proficiency in Python for data processing, scripting, and application development.
- Hands-on experience with one or more leading big data processing frameworks (e.g., Apache Spark, Apache Flink) and cloud-based data warehousing solutions (e.g., Snowflake, Amazon Redshift, Google BigQuery, Azure Synapse Analytics). Demonstrable experience with platforms like Databricks or similar is a strong asset.
- In-depth understanding and practical experience with data lake and lakehouse architectural patterns.
- Experience with various data storage solutions (e.g., relational databases, NoSQL databases, object storage like AWS S3, Azure Blob Storage).
- Solid understanding of ETL/ELT processes, data modeling principles, and data pipeline orchestration tools (e.g., Airflow).
- Basic knowledge of API development (RESTful APIs) and experience integrating data services.
- Strong SQL skills and experience with query optimization.
- Familiarity with version control systems (e.g., Git) and CI/CD practices.
- Excellent problem-solving, analytical, and troubleshooting skills.
- Strong communication and collaboration abilities.
Preferred Qualifications:
- Experience with other programming languages such as Java or Scala.
- Deep expertise in multiple cloud platforms (e.g., AWS, Azure, GCP) and their diverse data services.
- Knowledge of containerization technologies (e.g., Docker, Kubernetes).
- Experience with streaming data technologies (e.g., Kafka, Kinesis).
- Understanding of machine learning concepts and experience supporting MLOps.
- Contributions to open-source data projects.
We thank you for applying to this job position, we’re more than thrilled to start reviewing your profile and great skills! This is the first step in our selection process, so you will be hearing back from our awesome recruitment team regarding the next steps 😀
10Pearls Team
