Backend Software Engineer (Evals) – Support Automation Engineering

Who you are

You have experience as a Backend Engineer, particularly in ML/LLM-heavy domains, and are skilled in designing and building robust systems that measure quality and performance. You understand the intricacies of backend services and have a strong foundation in Python, enabling you to create reliable and extendable eval pipelines. Your experience includes working closely with cross-functional teams, particularly Data Science and Research partners, to ensure that the systems you build are effective and scalable.

You are familiar with continuous evaluation monitoring frameworks and have a knack for creating feedback loops that enhance system performance. Your technical expertise allows you to navigate complex challenges and implement solutions that drive impact across the organization. You thrive in collaborative environments and are eager to leverage cutting-edge AI models to solve real-world challenges.

Desirable

Experience with OpenAI technology or similar AI models is a plus, as is familiarity with automation products that empower teams. You are comfortable with rapid prototyping and have a focus on long-term quality and reliability in your work. Your ability to blend technical skills with a strategic mindset makes you a valuable asset to any team.

What you'll do

In this role, you will design eval pipelines that are reliable and reproducible, ensuring that the quality of OpenAI's support automation is consistently measured and improved. You will build the infrastructure necessary for continuous eval monitoring, including regression and drift monitoring, and create robust golden datasets that serve as benchmarks for performance. Your work will involve close collaboration with Data Science and Research teams to develop systems that not only meet current needs but are also extendable for future requirements.

You will be responsible for implementing feedback loops that enhance the automation products developed by the Support Automation team. This includes analyzing data and metrics to inform decisions and drive improvements in the systems you build. Your contributions will directly impact how knowledge is created, accessed, and applied across OpenAI, making your role crucial to the organization's success.

What we offer

At OpenAI, you will be part of a team that is dedicated to leveraging AI technology to improve operations and drive innovation. We offer a collaborative work environment where your ideas and contributions are valued. You will have the opportunity to work with cutting-edge technology and be at the forefront of AI advancements. We are committed to providing reasonable accommodations to applicants with disabilities and fostering an inclusive workplace culture. Join us in shaping the future of technology and making a meaningful impact in the world.