Application Guide

How to Apply for Task Development Engineer

at Model Evaluation and Threat Research

🏢 About Model Evaluation and Threat Research

Model Evaluation and Threat Research (METR) is at the forefront of AI safety, dedicated to rigorously evaluating the capabilities and alignment of advanced machine learning models. As a nonprofit project spun out of the Alignment Research Center, it offers a unique opportunity to directly influence the safe development of frontier AI. Working at METR means collaborating with a team of experts who are shaping how the world understands and mitigates AI risks.

About This Role

As a Task Development Engineer, you will design challenging evaluation tasks that push the boundaries of AI capabilities, ensuring these tasks remain difficult as models evolve. Your work directly informs the AI community about model limitations and safety, making your contributions critical to METR's mission. This role combines technical skill with creative problem-solving to build robust evaluations that can withstand the rapid advancement of AI.

💡 A Day in the Life

A typical day might involve brainstorming new task ideas with the team, prototyping a task in Python, and running it against a frontier model to test its difficulty. You might also spend time reviewing tasks from other engineers, providing feedback on their solvability and clarity, and iterating on your own tasks based on baseline results. Collaborating with researchers to align tasks with current AI safety concerns is also a regular part of the day.

🚀 Application Tools

Generate Cover Letter

AI-powered, tailored for Model Evaluation and Threat Research

Interview Prep

Practice questions for this role

🎯 Who Model Evaluation and Threat Research Is Looking For

  • Someone with a strong background in AI/ML, including hands-on experience with state-of-the-art models and an understanding of their strengths and weaknesses.
  • A creative problem-solver who can design novel tasks that test complex reasoning, creativity, or other high-level capabilities in models.
  • Detail-oriented with experience in quality assurance, ensuring tasks are solvable and free of unintended hints.
  • Proficient in programming (e.g., Python) to build task infrastructure and automate scoring, with a knack for optimizing workflows.

📝 Tips for Applying to Model Evaluation and Threat Research

1

Highlight any experience you have designing or contributing to evaluation tasks, benchmarks, or adversarial testing of AI models.

2

Showcase your ability to think like a model—describe a task you've created that specifically targeted a known limitation of current models.

3

In your resume and cover letter, provide concrete examples of how you've improved workflows or built tools to streamline development processes.

4

Research METR's published evaluations and mention specific ones you admire, explaining how you would extend or improve upon them.

5

Tailor your application to emphasize your commitment to AI safety and alignment, as METR is a mission-driven organization.

✉️ What to Emphasize in Your Cover Letter

["Your passion for AI safety and how your skills align with METR's mission of rigorous model evaluation.", "Specific examples of tasks or evaluations you've developed, detailing the challenges and your solutions.", 'Your technical expertise in AI/ML and programming, and how it enables you to create robust, scalable evaluation tasks.', 'Your collaborative spirit and ability to work with a distributed team, as the role is remote and requires cross-functional coordination.']

Generate Cover Letter →

🔍 Research Before Applying

To stand out, make sure you've researched:

  • → Read METR's published evaluations and blog posts to understand their methodologies and the types of tasks they've created.
  • → Familiarize yourself with the broader AI safety landscape, including other organizations like Anthropic, OpenAI, and ARC, to see how METR fits in.
  • → Study recent frontier model capabilities (e.g., GPT-4, Claude, Gemini) to understand current strengths and weaknesses.
  • → Look into the team members on METR's website to understand their backgrounds and potential collaborators.
Visit Model Evaluation and Threat Research's Website →

💬 Prepare for These Interview Topics

Based on this role, you may be asked about:

1 How would you design a task to evaluate a model's ability to perform long-horizon planning? Walk us through your process.
2 Describe a time you had to debug an evaluation task that was too easy or too hard. What steps did you take?
3 How do you stay updated on the latest AI capabilities and incorporate that into your task design?
4 What infrastructure improvements would you make to streamline task development at METR?
5 How do you ensure that your evaluation tasks are free from biases or unintended shortcuts that models might exploit?
Practice Interview Questions →

⚠️ Common Mistakes to Avoid

  • Don't submit a generic cover letter that doesn't mention METR or AI safety; show you've done your homework.
  • Avoid focusing solely on academic achievements without demonstrating practical application of your skills to evaluation tasks.
  • Don't underestimate the importance of infrastructure; METR values engineers who can build tools, so don't neglect to highlight your software development skills.

📅 Application Timeline

This position is open until filled. However, we recommend applying as soon as possible as roles at mission-driven organizations tend to fill quickly.

Typical hiring timeline:

1

Application Review

1-2 weeks

2

Initial Screening

Phone call or written assessment

3

Interviews

1-2 rounds, usually virtual

✓

Offer

Congratulations!

Ready to Apply?

Good luck with your application to Model Evaluation and Threat Research!

← Back to Job Listing Apply Now →