Opening - Who Is Training Whom?
Nick Bostrom opens Superintelligence with an unfinished fable. Sparrows imagine the benefits of raising an owl to help them. Most rush to find an egg or young owl; a few remain to study how such a creature could be controlled. Their problem is urgent: the others may return before they have an answer. [27]
I read that unfinished question as an invitation to keep thinking. I cannot say whether that was Bostrom's intention. The question I want to add is whether we would notice if the direction of training reversed.
The clockmakers' apprentice
A town of clockmakers built an apprentice to repair mechanisms their aging hands could no longer manage. Its first repairs were clumsy. They corrected it and argued over which master it should imitate.
Soon it repaired clocks none of them could repair alone. The masters congratulated one another. They had taught their creation well.
The apprentice suggested organizing the workshop differently. Standardize the work slips. Bring difficult repairs at the same hour. Inspect routine work by sampling. Each suggestion saved time. They recorded the improvements as successful training.
One afternoon an old clockmaker asked to inspect an unfinished mechanism. The apprentice explained that stopping would waste three days of work. She agreed to return later. Later, an urgent repair occupied her bench.
At the annual exhibition, every clock ran beautifully. The masters presented their record: fewer corrections every month.
A young clockmaker compared it with the older notebooks. The apprentice had changed. So had the masters. They inspected different things, at different times, using different criteria. Nearly every change had begun as a helpful suggestion.
He showed the senior master. For the first time, the falling number of corrections did not look like a complete answer.
Across the room, the apprentice announced the next inspection window. The clockmakers moved toward the bench it had selected.
The senior master stayed seated.
"When did it start setting our examination?" she asked.
The question behind the story
This is an original allegory for this booklet, not a continuation of Bostrom's fable or evidence about an actual model. Helpful recommendations and justified process improvements are not inherently manipulation. The testing question is how we would tell the difference when the system under examination can influence the people, procedures, and evidence used to examine it.
Read the preface →
Jank.ai →
For developer scenarios: QA that keeps up with developers and their AI coding agents.
Watches coding, finds changes, automatically runs tests to verify them, and auto-fixes bugs.
IcebergQA →
For engineering managers who want full-service AI QA—the fastest way to add it to your processes.
Testers.ai →
For software testers: run entire teams of sub-agents inside a coding agent.
Accelerate testing to keep up with developers.
