Research Scientist

The Center for Open Science (COS) is seeking a Research Scientist to facilitate the development of benchmarking whether LLM agents can evaluate, replicate, and independently conduct the research process. COS has received funds to conduct a 2-year research project in this emerging space.

The project will elaborate a conceptual framework for how the research process is conducted and evaluated (e.g., study design, conduct, analysis, interpretation) and different researcher personas (e.g., peer review, replication, and discovery) to develop specific tasks that can be benchmarked effectively and are amenable to testing with LLM agents. In collaboration with three partner groups with expertise in developing LLM agents and benchmarking, COS will refine the conceptual framework, assist with designing tasks, support developing and evaluating LLM agents to perform those tasks, assist with refining the tasks based on evidence, and releasing the framework to the wider research community to provide feedback. In addition to gathering feedback, this project aims to develop and organize other ways (e.g., competitions) for the research community to participate in the development, testing, and evaluation of LLM agents against these tasks.

About the role

The Research Scientist will play a central role in coordinating the work with our current collaborators and engaging the broader community to contribute to this project. If you are interested in contributing to the cross-section of emerging AI technologies with open science, then this is the opportunity you have been waiting for.

Additional Information

This is a fully remote position with a requirement to attend around 4 in-person meetings per year.

COS supports flexibility in work scheduling. Individuals who would like to work outside a 9am – 5pm Eastern Time (ET) schedule must work with their managers to establish core hours of availability. Modifications for different time zones will be considered for team collaboration.

Applicants should note that this position is associated with an exploratory project and will be grant-funded for a period of through and including December 31, 2026. Extensions of this period may be possible depending upon project evolution and business needs, as determined in COS’s sole discretion. Applicants are advised that the position may be eliminated at the conclusion of the initial grant period. Applicants are further advised that all employment with COS, including this position, is at-will, meaning that it may be ended at any time, with or without cause, and with or without notice by either COS or the employee. Due to the temporary nature of this position,

elimination of this position is not subject to COS’ Reduction in Force policy and associated severance.

Please apply by submitting a resume and answering the 5 qualifying questions in lieu of a cover letter. Answers to the qualifying questions should be succinct, please limit responses to approximately 250 characters. Questions about the position and COS are welcome and can be directed to jobs@cos.io. For information on COS, including more information on employee benefits and our company culture, visit our website at https://www.cos.io/careers.

Please note, in order to be considered for this position you must be authorized to work in the United States and have a U.S. address and bank account. Remote workers must be located within the U.S.

This position will remain open for applications until January 20, 2025. Interviews will begin at the end of January.

COS is an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, disability status, genetic information, protected veteran status, or any other characteristic protected by law. We also strongly encourage applications from members of groups underrepresented in science and technology industries.