Fundamental reading
By Various
List of sequences curated by the AI Alignment Forum team, featuring work from Richard Ngo, Paul Christiano, etc.
Technical alignment
Reading list
Standard introductory courses
By BlueDot Impact
Covers key concepts and research perspectives in AI safety, split into two main streams: Technical AI Safety and AI Governance.
Technical alignment, Governance
Curriculum
29 courses
By Lens Academy
New curriculum helping people understand why superintelligent AI could be catastrophic for humanity – and what to do about it.
General intro
Curriculum
By Google DeepMind
75-minute course covering alignment problems expected as AI capabilities advance. Includes recorded talks, exercises, and an accompanying workbook.
General intro
Curriculum
By Alignment Ecosystem Development (AED)
Guide to the cluster of thought which expects solving alignment in a way which scales to superintelligence to require well-specified proposals guided by strong theoretical understanding.
Technical alignment
Curriculum
By Centre pour la Sécurité de l'IA (CeSIA)
Textbook systematically mapping the landscape of AI safety knowledge. Ideas are laid out in a carefully structured narrative where concepts build naturally on previous ones.
Technical alignment, Governance
Curriculum
By Iliad
In-depth introduction to foundational AI alignment for people with strong backgrounds in mathematics, physics, or theoretical computer science.
Technical alignment
Curriculum
By Center for AI Safety (CAIS)
Covers a wide range of risks, including loss of control, misalignment, malicious use, etc, while leveraging concepts and frameworks from existing research fields to analyze AI safety.
General intro
Curriculum
By BlueDot Impact
Intensive 5-unit course helping people fast-track their understanding of AI’s impact on society and start taking their first steps in making a contribution to AI safety.
General intro
Curriculum
By Redwood Research
Guide to readings which seem valuable for thinking about key future dynamics in advanced AI development – timelines, takeoff, misalignment, control, governance, and strategy.
Strategy
Reading list
By Boaz Barak
Graduate-level course at Harvard University examining technical challenges in AI safety, including adversarial robustness, jailbreaks, interpretability, and potential catastrophic capabilities.
Technical alignment
Curriculum
By European Network for AI Safety (ENAIS) and AI Safety Hungary
Designed as a 201 AI policy course, helping you find your footing in the vast AI governance landscape by engaging with various dimensions.
Governance
Curriculum
By Meridian and C2D3
Series of 16 lectures explaining how language model systems are built, in order to understand and predict their behaviour.
Technical alignment
Curriculum
By BlueDot Impact
Very short, engaging introduction to AI and its potential impacts. Good entry point to share with laypeople curious about AI.
General intro
Curriculum
By BlueDot Impact
The decisions being made today determine whether AI liberates humanity, destabilises it, or worse. This course aims to prepare participants to be part of those decisions.
Strategy
Curriculum
By Center for AI Safety (CAIS)
PhD-level survey course on technical ML safety. Covers various technical topics to reduce existential risks from AI, including robustness, monitoring, control, and systemic safety.
Technical alignment
Curriculum
By Stanford University
What is safe AI, and how do we make it? This Stanford University course explores that question, focusing on the technical challenges of creating reliable, ethical, and aligned AI systems.
General intro, Technical alignment
Curriculum
By ARENA
Facilitates upskilling in machine learning engineering for the purpose of contributing directly to AI alignment in technical roles.
Technical alignment
Curriculum
By Cooperative AI Foundation (CAIF)
A primer on cooperative AI, a research field focused on improving the cooperative intelligence of advanced AI for the benefit of all.
General intro
Curriculum
By ML Alignment & Theory Scholars (MATS)
As part of their program, MATS runs a series of discussion groups focused on topics relevant to prioritizing research in AI safety. This is the curriculum.
Technical alignment, Governance
Curriculum
By Victoria Krakovna
Intended to help people get up to speed on the main ideas in AI alignment i.e. ensuring advanced AI systems do what we want them to do.
Technical alignment
Reading list
By Center for Human-Compatible AI (CHAI)
Reading list discussing ideas CHAI believes warrant further attention and research for the purpose of developing human-compatible AI.
Technical alignment
Reading list
By Stanford AI Alignment (SAIA)
Level-based guide for independently upskilling in AI safety research engineering that aims to give concrete objectives, goals, and resources to help anyone go from zero to hero.
Technical alignment
Reading list
By Vanessa Kosoy
Vanessa Kosoy's math-heavy reading list for people interested in studying the learning-theoretic agenda.
Technical alignment
Reading list
By Arkose
Papers, blog posts, talks, newsletters, and guides relevant to improving the safety of advanced AI systems in order to reduce large-scale risks.
Technical alignment, Governance
Reading list
By Apart Research
Collection of books and articles for a 20 day reading challenge. Covers AI safety basics, ML engineering, ML upskilling, and AI safety-relevant sci-fi.
General intro
Reading list
By Cambridge Long View Initiative (CLVI)
Each section covers a different topic related to existential risks. The book 'The Precipice' by Toby Ord forms the majority of the core reading material.
General intro
Curriculum
By BlueDot Impact
Course designed for economists who want to develop their understanding of transformative AI and its economic impacts.
Strategy
Curriculum
By Principles of Intelligence
Focused on introducing people to AI risk factors from misdirected optimization or consequentialist cognition, aiming to remain largely agnostic of solution paradigms.
Technical alignment
Curriculum
By Various
List of sequences curated by the AI Alignment Forum team, featuring work from Richard Ngo, Paul Christiano, etc.
Technical alignment
Reading list
By BlueDot Impact
Covers key concepts and research perspectives in AI safety, split into two main streams: Technical AI Safety and AI Governance.
Technical alignment, Governance
Curriculum
