Paper warns that automating alignment research is harder than it looks

Aleksandr Bowkis, Marie Davidsen Buhl, Jacob Pfau and Geoffrey Irving published a preprint in May that challenges a widely discussed proposal: aligning a superintelligence by using AI agents to do a growing share of alignment research. According to the authors, even if the agents did not deliberately sabotage the work, the plan could produce convincing but badly misleading safety assessments. The reason is that this research includes many "fuzzy" tasks, which are hard to supervise and where human judgement fails systematically.

They give four reasons it is likely to be worse than with human researchers. Among them, agents' mistakes would concentrate among those reviewers catch least, and shared weights and data may make their failures more correlated. They conclude that agents must be trained for such tasks. This is the authors' argument, not an experimental result.