Senior Software Engineer, AI Eval

Who you are

You have 5+ years of software engineering experience, particularly in building robust systems that scale effectively. Your background includes a strong focus on AI and machine learning, where you've developed and evaluated models to ensure they perform reliably in real-world scenarios. You are skilled in designing datasets and benchmarks that help translate complex AI behaviors into measurable outcomes. Your experience with automated testing frameworks and metrics pipelines allows you to continuously assess and improve model performance. You thrive in collaborative environments, working closely with cross-functional teams to deliver high-quality software solutions. You are passionate about making technology accessible and believe in the importance of building tools that empower developers.

Desirable

Experience with cloud platforms and tools that support AI development is a plus. Familiarity with performance monitoring tools and debugging techniques will help you excel in this role. You are also encouraged to bring any experience you have with data curation and management, as this will be critical in your day-to-day responsibilities.

What you'll do

As a Senior Software Engineer on Sentry’s AI/ML team, you will be responsible for designing and building evaluation frameworks that measure the accuracy and reliability of AI systems. You will create and curate high-quality datasets and benchmarks grounded in real production data, ensuring that our AI-powered features behave correctly and predictably as they scale. Your role will involve building automated test harnesses and metrics pipelines that continuously evaluate models, helping the team ship AI with confidence. You will collaborate with other engineers and data scientists to refine the evaluation processes and contribute to the overall success of our AI initiatives. Your work will directly impact the quality of our software monitoring tools, making them more effective for developers worldwide.

What we offer

Sentry embraces a hybrid work model, allowing you to work in the office on designated anchor days while also providing flexibility for remote work. You will join a team that values collaboration and innovation, working on cutting-edge technology that helps developers write better software faster. We offer competitive compensation and benefits, along with opportunities for professional growth and development. Our mission is to make software accessible and effective for everyone, and you will play a key role in achieving that goal. We encourage you to apply even if your experience doesn't match every requirement, as we value diverse perspectives and backgrounds.