Area: International AI agreements and government involvement in advanced AI development

Project Idea 1: Distributed training and covert training runs. Suppose the nations with the most advanced AI capabilities agree not to train models past a certain FLOP or capabilities threshold. In this world there may be inspections and other verification mechanisms to ensure that each nation is not using their large datacenters to break this agreement. However, there is the risk that a model above the agreed threshold could be trained using distributed training, and therefore not be caught by the verification and monitoring regime.

Key questions: 

• What is the current state of distributed training?

• How much compute and of which type (e.g., small clusters, gaming GPUs, CPUs??) could be recruited for a distributed training regime? And how large a model could be trained this way?

• What interventions might allow for the monitoring or prevention of this distributed training?

Project Idea 2: An AI agreement based on AI capability red-lines. The US and China may wake up to the risks from advanced AI; risks like misalignment, misuse, and threats to geopolitical stability. Centrally, China may be worried that the US will build powerful AI, and use it to totally disempower China. They may propose a regime where each nation is not allowed to develop AI systems which pass certain capabilities red-lines. 

Key questions:

• What are key red-lines? Do red-lines for preventing misuse and geopolitical instability also help with misalignment?

• How could nations verify this agreement? 

• Could automated AI auditors be used to verify that nations are not breaking agreements, without leaking information? How could these automated AI auditors themselves be created in ways that nations trust?

Project Idea 3: Institutions to carry over to a US AI National Project. The US government may wake up to the potential of advanced AI, and seek to create an AI National Project. This Project could involve the US government becoming much more involved in the activities of current AI companies, potentially in the form of a public-private partnership. We would like this National Project to have good safety practices and a robust, cautious safety culture. These may be ported over from the existing AI companies. 

Key questions:

• What are key safety practices to carry over to a US AI National Project from current AI companies? How could we ensure these are carried over?

• How might we ensure that the leadership of the US AI National Project cares about safety, potentially by keeping key individuals in oversight positions?

• What are key safety institutions and practices which are not present in current AI companies, but could be more viable as part of a US National Project?

I’m a researcher at the Machine Intelligence Research Institute, working on the Technical Governance Team. I’m excited about work on international coordination, and strategic considerations around the development of ASI.

I have a Masters degree in Physics, where I studied quantum optics and quantum fluids. I originally joined MIRI to work on technical AI alignment, and previously worked on AI safety as part of the first MATS cohort and at the Center for Human-Compatible AI.

Would be great for fellows to be reasonably self-directed, and have at least some experience with the field.

I’m excited to help scope and guide a research project that we are both excited about. I’m most excited about projects which are premised on AGI potentially coming soon, and involve governments becoming much more situationally aware. You can expect regular meetings and feedback on your project.

I think I am probably not best suited for mentoring projects where fellows would want significant assistance with coding. If someone was a strong coder and wanted help applying these skills to a technical governance project, that could be exciting.