Member of Technical Staff, LLM Evaluation

Who you are

You have a strong background in social sciences and machine learning, with a focus on analyzing natural language. Your experience includes developing evaluation methodologies and training classifiers, which will be crucial in assessing Copilot's real-world performance. You are a creative problem solver who enjoys collaborating with user researchers and product leaders to build automated evaluation frameworks. You understand the importance of user needs and are committed to ensuring that AI systems effectively support them.

You thrive in a team environment and are eager to contribute to a culture of inclusion and collaboration. Your analytical skills allow you to experiment with data collection techniques and implement methodologies that provide real-time insights into Copilot's effectiveness. You are passionate about leveraging technology to empower users and improve their experiences.

Desirable

Experience with automated evaluation frameworks and a deep understanding of user experience metrics would be advantageous. Familiarity with AI systems and their applications in real-world scenarios will help you excel in this role. A growth mindset and a commitment to continuous learning are essential as you navigate the evolving landscape of AI technology.

What you'll do

In this role, you will develop and implement cutting-edge methodologies to evaluate how well Copilot performs in various usage scenarios. Your responsibilities will include designing experiments to assess the effectiveness of AI systems, analyzing user interactions, and providing actionable insights to improve Copilot's performance. You will work closely with cross-functional teams to ensure that the evaluation frameworks align with user needs and business objectives.

You will be responsible for training classifiers and experimenting with different data collection techniques to enhance the evaluation process. Your work will directly impact how Copilot meets user needs, focusing not only on task completion but also on the affective aspects of the user experience. You will collaborate with product leaders to drive improvements based on your findings and contribute to the overall success of the Copilot initiative.

What we offer

Microsoft offers a dynamic work environment where innovation and collaboration are at the forefront. You will have the opportunity to work with cutting-edge technologies and contribute to projects that empower users globally. The company values respect, integrity, and accountability, fostering a culture where everyone can thrive. You will be part of a team that encourages growth and supports your professional development.

As a Member of Technical Staff, you will have access to resources and training that will help you expand your skill set and advance your career. Microsoft is committed to creating an inclusive workplace where diverse perspectives are valued, and you will be encouraged to share your ideas and insights. Join us in our mission to empower every person and organization on the planet to achieve more.