Data Platform Engineering Lead

Who you are

You have extensive experience in data engineering, particularly in architecting platforms that handle large-scale data flows — your expertise allows you to navigate complex challenges with petabytes of data and optimize costs effectively. You are a natural leader who thrives in guiding high-caliber teams, focusing on strategy and architecture rather than micromanagement. Your background includes working with machine learning engineers and data scientists to align technical roadmaps with AI objectives, ensuring that your team is well-equipped to meet ambitious goals.

You possess deep knowledge of cloud technologies, particularly AWS, and have hands-on experience with tools like Apache Flink and Spark — you understand the intricacies of data streaming and batch processing, and you are adept at implementing solutions that ensure low-latency and consistent feature serving. Your familiarity with lifecycle management tools, such as Apache Iceberg, enables you to drive innovation in data management practices.

You are passionate about building a strong engineering culture, advocating for a 'Platform as a Product' mindset — you believe in creating self-service tooling that empowers engineers and data scientists to deliver high-quality work efficiently. Your analytical skills allow you to monitor platform performance and reliability, using tools like Prometheus to ensure that service level objectives are met.

Desirable

Experience with FinOps practices is a plus, as you will be responsible for strategic financial operations related to data processing and storage. Familiarity with observability tools and metrics analysis will further enhance your ability to drive platform stability and performance.

What you'll do

In this role, you will lead a team of senior data engineers, focusing on the architecture and strategy of the Feature Platform and Streaming flows — your leadership will help remove roadblocks and foster a collaborative environment where innovation can thrive. You will partner closely with machine learning engineers and data scientists to ensure that the data infrastructure aligns with AI goals, driving the adoption of best practices in data management.

You will oversee the performance and cost-tuning of compute layers, ensuring that analytics are fast and efficient — your expertise in tools like EMR on EKS and Druid will be crucial in achieving these objectives. You will also be responsible for unifying online and offline data patterns, ensuring that your team can serve features consistently and reliably.

Your role will involve championing the use of Apache Flink within the organization, promoting its adoption to enhance real-time data processing capabilities. You will drive the maturation of the Apache Iceberg strategy, automating lifecycle management to streamline data operations.

What we offer

Unity offers a dynamic work environment where you can make a significant impact on the future of AI and data engineering. You will have the opportunity to work with cutting-edge technologies and lead a talented team of engineers. We encourage you to apply even if your experience doesn't match every requirement — your passion for data engineering and leadership will be valued here. Join us in redefining the landscape of data platforms and contribute to groundbreaking projects that shape the future of technology.