Senior Software Engineer (MLOps) – Annotation & Evaluation

Who you are

You have 5+ years of experience in software engineering with a focus on machine learning operations — you've designed and built systems that support the evaluation and deployment of AI models at scale. Your expertise in Python and familiarity with machine learning frameworks like TensorFlow and PyTorch enable you to create robust solutions for model benchmarking and evaluation. You understand the intricacies of data engineering and have experience working with data pipelines and telemetry systems. Your knowledge of containerization and orchestration tools such as Docker and Kubernetes allows you to deploy applications efficiently in production environments. You are comfortable collaborating with cross-functional teams, including data scientists and engineers, to drive trust and safety observability in AI products. You value a hybrid work culture that promotes collaboration and creativity, and you are eager to contribute to a team that is at the forefront of AI technology.

Desirable

Experience with AWS services and tools for machine learning, such as SageMaker, is a plus. Familiarity with MLflow for managing the machine learning lifecycle will enhance your contributions. You have a proactive approach to problem-solving and are always looking for ways to improve processes and systems.

What you'll do

In this role, you will design and build systems that automate the evaluation of AI models, including large language models (LLMs) and agents. You will lead efforts to develop benchmark suites and evaluation pipelines that incorporate trust and safety metrics. Your work will involve building and maintaining integrations with labeling systems, such as Label Studio, to streamline the dataset labeling process. You will collaborate closely with data scientists to ensure that the models are reliable and safe for production use. Additionally, you will drive the development of performance diagnostics tools that provide insights into model behavior and effectiveness. Your contributions will directly impact the quality and safety of Datadog's AI product offerings, ensuring they meet the highest standards of reliability.

What we offer

Datadog fosters a collaborative and inclusive work environment where creativity thrives. We offer a hybrid workplace model that allows you to balance your professional and personal life effectively. You will have the opportunity to work on cutting-edge AI technologies and contribute to projects that have a significant impact on our products and customers. We provide competitive compensation and benefits, along with opportunities for professional growth and development. Join us at Datadog and be part of a team that is shaping the future of AI in the tech industry.