Before we begin.

Preface - We Need More Than a Warning

The world feels a little crazy. AI keeps getting smarter. Listening to the argument around it, I am not always convinced the rest of us are keeping up.

This book is about testing for superintelligence and testing systems that may exceed our understanding, with urgency. It is not a general treatise on AI safety or a claim to have solved alignment. Safety provides the stakes and the context. The practical question is what we can observe, test, and do while our decisions still matter.

I also want to listen to the optimists and the skeptics. Perhaps progress will hit limits. Perhaps we will build systems that are extraordinarily helpful and remain under effective control. I hope so. But a credible possibility of irreversible catastrophe deserves serious testing even when it is far below fifty-fifty. A hypothetical one-in-a-hundred chance of losing humanity would not reassure me. It would put this at the top of my testing priorities. Chapter 6 examines the arguments for optimism and limits; Chapter 27 separates the probability claims from the risk decisions we need to make.

I keep hearing about the things a sufficiently capable AI could do to us. Some accounts end with human extinction. Others end with an instruction to stop building, switch off the computers, or put the system in a box. As a software tester, I keep waiting for the next part of the conversation: how would we know what the system can do, what it is doing, and whether the protections actually work?

The alarm is loud. I hear fear, anger, and frustration. What I hear much less often is a concrete proposal for testing the thing everyone is worried about. What would we measure? What would count as evidence? How would we notice that the situation had changed?

I reached out to several prominent people warning about AI and asked whether they had thoughts on testing for AGI and what comes after it. I got nothing back. To be fair, these were direct messages, not scheduled conversations with people I knew well. I do not know whether they saw them. Silence in my inbox does not establish that they have no answers. But it left my question unanswered.

That is part of why I am writing this book. Perhaps a book can start a conversation that a DM could not. If you are warning that these systems could destroy the world, I want to talk with you about how we would test for the capabilities and behaviors behind that warning.

Why a software tester is asking

I have spent years working on testing at scale, testing AI, and using AI to test software. I was a test manager on AI-based search systems at Microsoft, in the work that became Bing, and on Chrome at Google. These problems were part of my working life long before the current enthusiasm for large language models.

In my recollection of the search competition, Google seemed to have all the smart engineers. Neural networks felt like a practical way for us to compete: let a learned system find improvements we could not keep handcrafting. What felt like a hack became more capable over time. For those of us testing it, that meant learning how to examine a system whose behavior was not simply a list of rules we had written.

That responsibility extended beyond examining model outputs. My team was also accountable for the infrastructure used to train models, evaluate them, and run evaluations again. We examined statistics about the training data and how effectively people were using that infrastructure. We did basic analysis of training configurations and model hyperparameters to understand whether the choices people were making were helping. Testing meant looking at the machinery and the work practices that produced the result, not only the result at the end.

We also worked on data quality, including the human judgments used for training. In one case I remember, a vendor responded to disagreement between raters by firing one of them. That practice was distorting our training data. I return to it in the testing-foundations chapter because it captures a larger problem: apparent agreement can improve while the process producing it becomes less trustworthy.

I later cofounded test.ai, before the current LLM wave, to apply AI to software testing. Google's AI-focused venture fund, Gradient Ventures, led its 2018 Series A. [26] Testing AI and building AI that tests have been connected problems throughout my work.

So yes, I sometimes find myself arguing with a podcast or a conference presentation. I hear a familiar testing failure presented as a new mystery: trusting an answer key nobody checked, confusing a score with coverage, or accepting a demonstration as evidence of reliability. Software testers have spent years encountering these patterns, learning which techniques help, and finding out where they fail.

We have also spent years thinking about oracles: how to judge a result when the correct answer is difficult to specify. A research benchmark and a production testing program face different demands. Production adds changing environments, messy inputs, integrations, users, and failures that escape the test environment. I want more of that experience in this conversation.

This is not a claim that frontier researchers never test or that existing testing practice solves superintelligence. It is a request to bring the accumulated experience of software testing into the work, then discover precisely where it stops being enough. I do not hear enough public discussion of this less glamorous work: testing the training infrastructure, examining data quality, and investigating the incentives behind the labels. That does not establish how much of it happens inside labs. It does mean I want to see the evidence, rather than assume someone has covered it.

The loneliest testing conversation

At a conference, between sessions, I have tried describing this to other testers: this may be the hardest testing problem we have ever faced. People agree that it sounds hard, and then the conversation moves back to prompts for testing a website. That work matters. I still find the contrast jarring.

I have felt underwhelmed, and sometimes lonely. Writing this is partly cathartic. I want more people in our own profession to spend time thinking about the problem, turning concerns into experiments, and challenging the experiments. The research cited in this book shows that meaningful work exists. My frustration is with the distance between that work, the urgency of the claims, and a testing practice we can actually run.

I sat down with my fourteen-year-old son and asked him to help invent prompts: how could we tell whether something was smarter than us? We tried questions in ChatGPT and read the answers. Some sounded to me like pageant answers: polished, reassuring, and difficult to learn much from.

That did not tell us whether the system was being deceptive or simply producing the kind of response it had learned to produce. Then I asked him, "What if it was lying to us? How could we detect that?"

His eyes got big.

That is the question I want us to stay with. Not because every reassuring answer is a lie, but because our test has to distinguish the possibilities. We need to turn that question into instruments, controlled experiments, and continuously running checks.

This could be the most consequential testing work we ever do. Perhaps, in a sense none of us wants, it could be the final test. I would rather sound impatient now than discover later that we never built the means to notice. We needed to start yesterday.

A warning needs a test

I do not want our testing strategy to depend on everybody agreeing to stop. That is a planning position, not a prediction that policy cannot change anything. Pauses, restrictions, and technical controls may matter. We still need methods that work while capable systems exist and people continue trying to use them.

A sandbox is not an answer to all those questions. It is something else to test. An air gap removes important connections, but it does not establish that every physical or human communication channel is absent. A model that helps build its own protections has not independently certified them. And a promise that a system will behave safely is an output to examine, not a reason to stop examining.

We should take those concerns seriously without pretending we have proved every boundary will fail. The practical question is what a control prevents, under which assumptions, and what evidence would reveal its failure. This book will return to that question often.

The test is allowed to be wrong

The evals bother me too. We build an exam, give it to a model, and debate the percentage. Meanwhile the questions may be ambiguous, the reference answers may be wrong, the scoring software may have bugs, and the environment may be helping the model in ways we did not intend. Documented benchmark reviews show why these are real testing questions. [3, 4]

Some evaluations go far beyond multiple-choice questions. That is progress. But complexity does not make an evaluation trustworthy. We must investigate how it works and how far its conclusions reach.

Now add a more difficult possibility: the system gives us an answer we could not have created. Sometimes we can still check it. Sometimes checking it is itself a research project. Sometimes we may not even recognize the question we should have asked.

Begin with today

After the opening allegory, the argument starts with documented work, not a declaration that superintelligence has arrived. It follows mathematics beyond familiar exams, learning beyond human examples, and AI systems connected to laboratories, robots, and simulated worlds. At each step, the useful capability brings a new testing question.

A fictional research assistant connects the chapters. It passes an exam, produces an unfamiliar result, asks for tools, acts, and eventually challenges our ability to judge and observe it. This is a teaching scenario, not a claim that one current product already combines every capability described here.

Think of a gorilla watching people build a satellite. It could make accurate observations and still miss the purpose of almost everything happening in front of it. The analogy is limited, but the discomfort is useful. Testing does not grant us a complete view just because we are the ones holding the clipboard.

If we wait until a system is unmistakably beyond us to invent the tests and instruments, the time we need to prepare may already be gone. We should start while we can still discover our mistakes in smaller, more recoverable settings.

Why the owl

The owl is a deliberate homage to Nick Bostrom's Superintelligence: Paths, Dangers, Strategies. His book explores routes to superintelligence, takeoff dynamics, and the control problem. It also discusses monitoring, boxing, and tripwires; it would be unfair to say it ignores detection and control. [24]

The dedicated deception and safety sections also examine researchers who are already working on these problems. My frustration with public discussion should not erase that work.

The question I want to bring into the foreground is the working tester's question: how do we recognize a consequential capability, challenge the claim, and know whether our observation and control mechanisms are working? The homage is an acknowledgment of that earlier work, not an endorsement or collaboration.

If superintelligence could destroy the world, I would like us to be able to detect the relevant changes before they become irreversible, recognize what is happening while it happens, and, hopefully, still be here to examine the evidence afterward. The stakes make testing more urgent, even when it cannot give us certainty.

This is a developing booklet. Every chapter is a working draft. Executed examples are distinguished from proposed experiments. As the work grows, claims should acquire better evidence, examples should become reproducible, and weak arguments should change. A book about testing ought to make it possible to test what it says.