Contractor, Technical Benchmarking Lead / Researcher

Core responsibilities:

• Lead the technical development of LLM evaluation benchmarks and coordinate with key team members
• Design and develop benchmark architectures, evaluation rubrics, and scoring methodologies independently
• Apply LLM-as-judge frameworks and system prompt engineering in evaluation pipeline design
• Conduct statistical analysis of model outputs to validate benchmark effectiveness
• Contribute to technical research writing and documentation

This role contributes to the development of robust evaluation frameworks for large language models, supporting ethical AI development and policy through technical benchmarking research.

Required skills and qualifications:

• Experience with LLM evaluation benchmarks and technical development
• Ability to design benchmark architectures, evaluation rubrics, and scoring methodologies
• Knowledge of LLM-as-judge frameworks and system prompt engineering
• Statistical analysis skills for validating benchmark effectiveness
• Technical research writing and documentation capabilities

No compensation, perks, or culture highlights mentioned in the posting.

New York University's Center for Mind, Ethics, and Policy focuses on research at the intersection of cognitive science, ethics, and policy, particularly regarding emerging technologies like AI.