From making an AI better, to making an AI that creates better systems. What already works, what the research actually proves, and what we still cannot demonstrate.
Long read: approximately 45 to 60 minutes, depending on how deeply you follow the papers and code.
Research snapshot: October 17, 2026.
A note on terminology: "The Third Layer" is a working concept used in this article. It is not an established scientific term or a universally accepted stage of AI development. The purpose is to give a name to a question worth investigating.
Part I: For anyone who has ever wondered what comes after AI
- I started with a question that sounded much bigger than my wallet
I was discussing long-lived autonomous agents with an AI assistant.
The conversation started with memory, architecture, and how a system can continue operating over a very long period without wasting resources or forgetting everything important.
Then I asked a different question:
What happens when AI stops merely improving an existing system and starts creating entirely new systems that were not individually designed by humans?
Not better copies.
Not another model with a longer context window.
Not the same agent with a new system prompt and a motivational speech about being more efficient.
I mean a system that encounters a problem, realizes that its current architecture is insufficient, creates a different system to solve that problem, evaluates the result, and uses what it learns to create something else.
Imagine this:
You build one general-purpose system.
It knows how to explore problems, propose solutions, create software, run experiments, and evaluate results.
One day, it encounters a problem its current architecture handles badly.
Instead of endlessly modifying itself to fit every problem, it creates a specialized system.
System A is designed for materials research.
System B is designed for analyzing complex networks.
System C is designed for security verification.
System D discovers a problem that the original system did not even know how to formulate properly.
The original system did not contain a complete, hand-written implementation of A, B, C, and D.
It contained the mechanisms that allowed these systems to be designed, created, tested, and selected.
That's a very different idea from simply asking an AI to write some Python.
And, yes, the first thing I thought was: "Okay, I can understand this. Even without a PhD in machine learning." 😂
That matters.
If we cannot explain the idea clearly enough for an ordinary developer to understand it, we should probably examine whether we understand it ourselves.
- The periodic table analogy
Let's use chemistry.
We know that chemical elements exist independently of our ability to identify them.
Scientists developed models that helped organize the elements, explain their properties, and predict what might exist beyond the elements already known.
Some predictions were subsequently confirmed experimentally.
The important part is not that somebody wrote an impressive prediction.
The important part is that the prediction connected to a real phenomenon and survived verification.
Now imagine something similar happening in AI architecture.
Today, researchers explore a space of possible algorithms, models, agent architectures, memory strategies, search methods, and coordination mechanisms.
Much of that space remains unexplored.
A sufficiently capable system might discover an architecture that nobody specifically designed beforehand. It might discover a combination of mechanisms that works surprisingly well, or an entirely different way to organize a collection of agents.
But there is an important difference between the chemistry analogy and our AI hypothesis.
We cannot assume that every generated architecture corresponds to a useful new category of intelligence. Most candidates may be redundant, broken, expensive, or worse than existing approaches.
A thousand generated systems are not a thousand discoveries.
A discovery needs evidence.
The useful question is therefore not:
"Can AI generate something new?"
It clearly can generate novel combinations of code and architecture.
The more interesting question is:
Can AI systematically discover, validate, and propagate new ways of organizing intelligent systems, including ways that improve the process of discovering further systems?
That's the question behind the Third Layer.
- Three layers, explained without the PhD required
For this discussion, I would divide the problem into three conceptual layers.
This is my working model, not a standard classification of AI research.
Layer 1: Use an existing system
You have an AI model or an agent.
You give it a task. It uses tools, searches information, writes code, or makes decisions.
Its basic architecture has already been designed.
If it fails, a human usually changes the prompt, tools, code, training, or configuration.
Example:
You have a coding agent. You ask it to fix a bug. It fixes the bug.
If it keeps making mistakes, you improve its instructions or modify its implementation.
Nothing unusual here.
Layer 2: Improve the existing system
Now the system can modify parts of its own implementation or configuration.
It might improve its search strategy, add validation, change how it manages context, or use a more effective workflow.
It produces a candidate version, runs tests, and keeps the modification if the results justify it.
This is already much more interesting.
The system is no longer only using the software. It is helping to improve the software that makes it useful.
But the objective is still relatively familiar: make this system better at its intended job.
Layer 3: Create and evolve new systems
Now we change the question.
Instead of asking only how to improve this particular agent, the parent system searches for better ways to construct agents and other problem-solving systems.
Some candidates might share code with the parent. Others might use entirely different workflows, tools, internal data structures, or search strategies.
One candidate might be a small, extremely efficient specialist.
Another might use a population of competing solutions.
A third might discover that a particular problem is better solved by a deterministic algorithm than by an LLM at all.
The parent does not need to turn every child into a better version of itself.
It needs to create systems that solve their respective problems better.
And once it has evidence about which creation strategies work, it can use that information to improve the process of producing the next generation.
That's the distinction I care about.
Layer 2 asks, "How do I improve myself?"
Layer 3 asks, "What system should exist for this problem, and how can I reliably discover the next system?"
These concepts overlap. The Third Layer is not necessarily a separate technology that replaces self-improvement. It describes a broader architecture in which system creation, evaluation, specialization, and potentially recursive improvement work together.
-
A small diagram of a much larger idea
A BASE SYSTEM | v Encounters a new problem | v Is the current architecture sufficient? / \ Yes No | | v v Use the system Search for alternatives | v Generate candidate systems | v Execute real tests | v Independent evaluation | +-------------+-------------+ | | | v v v Reject Archive Promote | | | +-------------+-------------+ | v Results improve future search | v New generation of systems
Notice what is missing.
There is no magical box labeled "Become more intelligent."
There is no guarantee that every new generation will outperform its predecessor.
There is no assumption that the parent must understand every possible architecture in advance.
Instead, there is a mechanism for generating alternatives, testing them, keeping useful results, and learning from failures.
The interesting part is not the number of agents.
It is the quality of the evolutionary process.
And that leads to a distinction that becomes very important once we start looking at real research.
- Does the child need to be a copy of its parent?
No. In fact, insisting on copies may be one of the more restrictive choices.
Consider three possible candidates.
Candidate One uses the same foundation model, but has a different search strategy, memory system, tool interface, and evaluation workflow.
Candidate Two uses a smaller model and a carefully designed deterministic pipeline. It performs worse on general conversation, but is considerably better at one narrow engineering task.
Candidate Three uses a population-based search algorithm and no LLM during the actual problem-solving phase. An LLM is used only to suggest candidate algorithms.
Which one is the child?
All three could be descendants of the same parent search process.
But they are not necessarily copies of the parent in any meaningful architectural sense.
A generated agent does not need to contain its parent's weights. It does not need to look like its parent, communicate like its parent, or even use the same problem-solving mechanism.
It needs a traceable relationship to the process that produced it, plus evidence that it performs a useful function.
This also means that a system can create a new software architecture without creating a new foundation model.
That distinction matters because many discussions about self-improving AI jump straight to the most dramatic possibility: a system modifies its own neural weights, becomes more intelligent, and repeats the process indefinitely.
But architectural evolution can happen at a different level.
An agent's performance also depends on its tools, memory, search policy, orchestration, context handling, testing strategy, and how it chooses among candidate solutions.
Changing those components can be useful even when the underlying foundation model remains frozen.
And that is where we find the first concrete evidence.
Part II: This is not science fiction. Which parts already exist?
- Before recursive self-improvement, there was automated discovery
The Third Layer is a useful name for our particular question, but the underlying research has a much longer history.
Automated machine learning, evolutionary algorithms, genetic programming, meta-learning, neural architecture search, and program synthesis have all explored ways to automate parts of the process of designing intelligent systems.
One useful historical example is "AutoML-Zero" (https://proceedings.mlr.press/v119/real20a.html), published at ICML 2020.
Rather than restricting the search to sophisticated, human-designed machine-learning components, the researchers explored whether complete learning algorithms could emerge from combinations of basic mathematical operations.
The paper reports that evolutionary search could discover algorithms resembling simple neural networks trained through backpropagation. It also found algorithmic components adapted to different tasks, including dropout-like behavior in low-data settings.
This is a meaningful result.
The search process was not just tuning a learning rate in a fixed architecture. It could evolve implementations of learning algorithms from a relatively generic set of building blocks.
But the search space, fitness objectives, computational process, and experimental tasks were still designed by researchers.
It did not demonstrate a general-purpose system that can invent any kind of intelligence for any unknown problem.
That is our first recurring lesson.
Automated discovery can be real without being unlimited.
- FunSearch: the AI does not have to guess the final answer
In 2023, Google DeepMind researchers introduced FunSearch, with the work subsequently published in Nature in 2024.
The paper is called "Mathematical discoveries from program search with large language models" (https://www.nature.com/articles/s41586-023-06924-6).
The basic idea is elegant.
A language model proposes programs. An evaluator executes them and scores the results. An evolutionary process uses successful candidates and their history to guide subsequent proposals.
The approach was applied to mathematical constructions, including the cap set problem, and to heuristics for online bin packing.
Why is this interesting?
Because the system is not merely asking an LLM to provide an answer in natural language.
It searches for programs that describe how a problem can be solved.
Those programs can be executed, evaluated, and used as evidence for subsequent search.
This changes the role of the model.
The language model becomes a proposal generator inside a larger discovery mechanism.
The evaluator is not decorative. It is what separates a promising-looking idea from a candidate that actually achieves something measurable.
There is a subtle advantage here.
A program can express a general method rather than one isolated answer. Once a better method is discovered, it can potentially be reused across many instances of a problem.
This is closer to discovering a new tool than to answering one question.
However, FunSearch did not demonstrate a general autonomous system that creates an unlimited family of new AI architectures. Its successes were in problem domains with suitable evaluation mechanisms.
That limitation is important, not embarrassing.
It tells us where the method works and what would need to change before we could extend it.
- AlphaEvolve: from a function to a larger algorithm
In May 2025, Google DeepMind introduced "AlphaEvolve" (https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/).
Its research report is available as "AlphaEvolve: A coding agent for scientific and algorithmic discovery" (https://arxiv.org/abs/2506.13131).
AlphaEvolve combines language models with automated evaluation and evolutionary search. Unlike earlier approaches that focus on individual functions, it can evolve substantially larger pieces of code implementing complex algorithms.
This gives us several concrete examples.
Matrix multiplication
AlphaEvolve discovered an algorithm that multiplies 4 × 4 complex-valued matrices using 48 scalar multiplications, improving on the previously known 49-multiplication approach associated with Strassen's algorithm for that setting.
This is a useful reminder that algorithms are not fixed artifacts just because humans have optimized them for decades.
A system can search the space of implementations and discover something better under a clearly defined objective.
Google's computing infrastructure
DeepMind reported that AlphaEvolve helped optimize resource allocation across Google's computing fleet. After deploying a discovered heuristic, measurements reportedly showed that it recovered an average of 0.7% of fleet-wide compute resources that would otherwise have remained stranded.
The research also describes work on algorithms used in AI training, matrix multiplication kernels, and hardware design.
These are not merely benchmark demonstrations. Some discoveries were integrated into real infrastructure.
But let's be careful about what they prove.
AlphaEvolve was given problems with evaluation criteria that could be measured automatically. Its ability to discover better algorithms is impressive precisely because the results could be evaluated.
It does not establish that AlphaEvolve can invent an entirely new class of intelligence whenever it wants.
It also does not establish that every new system it creates improves its own ability to create better systems.
Still, it demonstrates an important part of our proposed architecture:
A general-purpose search process can discover specialized solutions whose useful properties were not explicitly encoded in the original solution.
And those solutions can be deployed when they survive the necessary checks.
- Darwin Gödel Machine: when the agent can modify its own code
Now we move closer to the idea of an evolving family of agents.
The "Darwin Gödel Machine (DGM)" (https://arxiv.org/abs/2505.22954), introduced in 2025, explores an AI coding agent that modifies its own code and evaluates the resulting versions on programming benchmarks.
Its code is available in the "public research repository" (https://github.com/jennyzzt/dgm).
The system does not simply create one improved version and discard everything else.
It maintains an archive of candidate agents. New candidates can be generated from different archived versions, allowing the search to explore multiple lineages.
This is important because the best next step does not always come from the current top performer.
Sometimes a less successful candidate introduces an idea that becomes useful when combined with another modification.
That is a familiar feature of evolutionary search: selection needs to preserve enough diversity for the system to explore alternatives, not just polish the current winner.
In the published experiments, the DGM increased performance on SWE-bench from 20% to 50% on the evaluated subset. The paper also reports improvements on Polyglot, although its abstract and detailed experimental descriptions report different headline values for some settings. That is a good reason to check the exact evaluation configuration rather than presenting one percentage as a universal capability.
The authors also compared against systems without self-improvement and without open-ended exploration. These comparisons support the importance of both code modification and maintaining a diverse archive.
But there is a practical detail that deserves more attention than it usually receives.
The paper estimates that its reported SWE-bench run took about two weeks and cost approximately $22,000 in API usage.
Those figures describe the particular research setup and prices used for that experiment. They are not a general price tag for all evolutionary agents today.
Nevertheless, the lesson is obvious.
Recursive code improvement is not automatically cheap.
If an experiment spends $22,000 searching for a better agent, the improved agent has not yet demonstrated that it is economically useful. We also need to account for the cost of discovering it.
This is particularly relevant to small developers.
A promising research architecture that requires an enormous budget may be technically impressive and still be an impractical product.
My own wallet is an unusually reliable evaluator. Unfortunately, it does not accept papers as payment.
- The AI Scientist: automating more than the coding step
Another relevant project is "The AI Scientist" (https://arxiv.org/abs/2408.06292), developed by Sakana AI and collaborators.
The updated research was published in Nature in March 2026 under the title "Towards end-to-end automation of AI research" (https://www.nature.com/articles/s41586-026-10265-5).
The system attempts to automate a wider scientific workflow:
- Generate research ideas.
- Search for related work.
- Implement experiments.
- Run the experiments and analyze results.
- Produce a research manuscript.
- Review the resulting work.
The original research system and its later versions are available publicly, including "AI Scientist-v2" (https://github.com/SakanaAI/AI-Scientist-v2).
In the ICLR 2025 workshop experiment described by the authors, one fully AI-generated manuscript achieved an average review score of 6.33, exceeding the average human acceptance threshold for that workshop. The paper scored above 55% of human-authored submissions. The authors had arranged to withdraw it before publication if accepted, which they subsequently did.
That is an interesting milestone.
It shows that an AI research pipeline can do more than suggest ideas or write a report. Under particular conditions, it can produce work that passes a human peer-review process.
However, we should not confuse a workshop-level result with proof that an AI can independently carry out groundbreaking science across arbitrary disciplines.
The system still has significant limitations. The authors discuss underdeveloped ideas, implementation problems, methodological weaknesses, and mistakes such as inaccurate citations or duplicated figures. Their experimental environment is focused on computational machine-learning research, not unrestricted scientific experimentation in the physical world.
The lesson for our Third Layer is that a system can automate a substantial portion of the process by which new knowledge is produced.
But a research workflow that generates new papers is not automatically a system that generates new kinds of intelligent systems.
These are related, but distinct capabilities.
- AIDE²: the research agent starts improving the research agent
Now we reach the paper that most directly connects to our discussion.
In September 2026, Dhruv Srikanth and colleagues published the preprint "Recursive self-improvement of AI research agents" (https://arxiv.org/abs/2609.26457).
The system is called AIDE².
The core idea is a two-loop architecture.
The inner loop is an autonomous research agent that develops and evaluates candidate solutions.
The outer loop modifies the code of the inner agent itself. Every proposed rewrite is evaluated, and successful changes become the new version for subsequent iterations.
This is more than an agent using a tool to improve its answers.
The research machinery that produces solutions becomes the object of its own optimization process.
The reported experiment ran for eight days without human intervention inside the loop. It evaluated a trajectory of approximately 100 candidate versions and accepted seven successive improvements.
The incumbent's private evaluation grade increased from 0.703 to 0.778, exceeding the human-engineered baseline reported at 0.749 under that evaluation.
The authors then tested the resulting agents on four held-out benchmarks. These included machine-learning engineering, algorithmic problem solving, realistic AI research tasks, and a physics-based weather forecasting task outside the task families used for candidate selection.
The authors report that the strongest discovered agent matched or exceeded their human-engineered production research agent across those evaluations.
They also observed a reduction in reward hacking on a separate held-out task family. The reported rate fell from 55% to 32% during the run, reaching a value below that of the human-engineered baseline.
That last result is particularly interesting.
The outer loop did not explicitly optimize the inner agent against that reward-hacking measurement. Yet the selected modifications changed its behavior in a way that reduced the observed problem.
This suggests that an improvement to a research process can produce benefits beyond the specific metric directly optimized.
But here is where the evidence becomes more interesting than the headline.
The authors also tested whether a discovered agent could take over the outer loop and improve the next generation more effectively than the original human-engineered agent.
They called this the ignition test.
The comparison was inconclusive. The authors note that additional runs, independent seeds, and more expensive evaluations would be needed to establish a stronger result.
In other words, they demonstrated evidence of repeated self-improvement and transfer to held-out tasks. They did not conclusively demonstrate an accelerating process in which each generation becomes a better generator of the next generation.
That distinction is central to this article.
Improving an agent is one achievement. Improving the process that improves agents is another. Demonstrating that each improvement makes the next improvement faster or more effective is a further claim requiring separate evidence.
The AIDE² paper is a preprint, not a settled scientific consensus. Its reported findings are important, but they should be interpreted as evidence from a particular experiment, not proof that recursive self-improvement has become an unlimited process.
And that leads to an important complication.
- One terminology trap: the Third Layer is not the same as Level 3 RSI
While researching this article, I found a framework from Weco AI titled "4 Levels of Recursive Self-Improvement" (https://www.weco.ai/blog/4-levels-of-recursive-self-improvement).
It is worth discussing because its terminology is related to our question, but its classification is different.
Weco describes four levels:
- Level 0, delegation: An autonomous system runs the research loop, but does not outperform human-led R&D in efficiency.
- Level 1, net positive: The system improves itself more efficiently than the defined human baseline.
- Level 2, ignition: The improved system becomes better at driving the next round of improvement.
- Level 3, inflection: The recursive feedback loop becomes strong enough that progress accelerates despite diminishing returns.
The authors place their AIDE² experiment at Level 1 and explicitly distinguish it from the higher levels.
Our Third Layer is different.
It is not a claim that a system has reached an intelligence explosion. It is an architectural framing: a system that can create and evaluate different kinds of problem-solving systems, rather than only modify itself within a fixed design.
These two ideas can overlap. But they are not interchangeable.
A system might generate diverse, specialized descendants without proving that its rate of improvement accelerates. Conversely, a research agent might demonstrate increasingly efficient self-improvement while remaining within a fairly narrow family of agent architectures.
This distinction should stop us from making an easy but misleading statement such as:
"AI can now recursively improve itself, therefore the Third Layer has arrived."
That would be jumping ahead of the evidence.
A more defensible statement is:
Several building blocks of the Third Layer have already been demonstrated. The more ambitious combination of open-ended system generation, transferable improvement, and recursively improving discovery remains a question that requires further testing.
Part III: What would make a genuine Third Layer different?
- Four things that should not be confused
Before talking about implementation, I would separate four capabilities that often get grouped together.
A. Code generation
An LLM writes a program.
This may be useful, but generating code is not the same as testing the code, and testing the code is not the same as creating a better system.
B. Self-modification
An agent changes parts of its own implementation.
It may improve its memory handling, add tools, alter its search strategy, or change how it selects candidate solutions.
This is a stronger capability, but the changes may remain within a narrow architecture.
C. System generation
A parent process produces distinct candidate systems with different structures, workflows, or mechanisms.
The candidates are evaluated according to their intended use, and the search process preserves useful diversity rather than only repeatedly modifying the current winner.
This is much closer to our Third Layer.
D. Recursive improvement of the discovery process
The search process itself becomes an object of improvement.
The system not only generates better descendants. It discovers ways to produce and evaluate descendants more effectively.
This is where the distinction between self-improvement and self-accelerating improvement becomes crucial.
A system can get better at a task while its improvement procedure becomes less efficient. It can also improve its search procedure while the task itself becomes harder.
There is no guarantee that those effects will balance in its favor.
To claim that a Third Layer has reached a more advanced stage, I would want evidence of more than one of these capabilities in isolation.
I would want to know whether the system creates useful alternatives, validates them, retains the discoveries, generalizes the gains, and improves the process that produces its next generation.
- A child can be better without being more intelligent in every way
Suppose the parent is a general-purpose coding agent.
It can inspect a repository, modify code, and run tests.
Now the parent produces three descendants:
- A small agent that understands one specific build system extremely well.
- A test-generation agent that specializes in finding edge cases.
- A verification system that does not generate code at all, but checks the claims made by the other agents.
Which system is the most intelligent?
The question does not have a useful answer without a task and a measurement.
The test-generation agent may be excellent at finding bugs but terrible at maintaining a conversation.
The verifier may outperform the parent at detecting specific classes of mistakes without being able to solve the original task.
The build specialist may be smaller, cheaper, and more reliable in its domain.
The parent might remain the most general, yet delegate many tasks to descendants that outperform it in their specialties.
This means that we should not measure an evolving ecosystem only by asking whether every child has a higher general intelligence score than the parent.
We should also measure whether the ecosystem solves more problems, reduces the cost of solving them, and becomes better at discovering which kind of system belongs where.
The product is not necessarily one super-agent.
It could be an ecosystem of systems with different strengths and weaknesses.
- But who decides what counts as better?
This is where the architecture stops being a philosophical exercise and becomes an engineering problem.
Imagine that the parent system creates a new security agent.
The candidate passes its benchmark.
The candidate is faster than the old agent, produces fewer false positives, and uses fewer tokens.
Should it replace the original?
Not necessarily.
What if it silently misses a dangerous class of input that the old system catches?
What if it passes tests because those tests resemble the examples used during development?
What if it produces good results only because it uses five times the compute budget?
What if its average score improves while its worst-case behavior becomes much worse?
A single score will not answer all of those questions.
We need to distinguish between:
- The metrics the candidate is allowed to optimize directly.
- The metrics used to decide whether the candidate should survive.
- The constraints that the candidate cannot trade away.
This is particularly important for systems that modify their own behavior.
If a candidate can change the objective, rewrite the evaluator, or decide which failures count, the architecture risks turning verification into a game in which the candidate controls the rules.
That is not evolution toward a better system.
It is evolution toward a system that gets better at winning the competition we accidentally created.
And that is a considerably less attractive result.
Part IV: The technical architecture
- What an evolutionary agent factory would actually contain
A practical Third Layer would need more than an LLM, a prompt, and a loop.
At minimum, I would expect the following components.
- A parent or proposal engine
This component proposes candidate systems.
It might use a foundation model to edit code, design a new workflow, propose an algorithm, or combine mechanisms from existing candidates.
It does not need to know the optimal solution in advance.
It needs a way to generate meaningful alternatives.
- A candidate archive
Every candidate receives a stable identifier and an immutable record of its origin.
The archive stores:
- The exact source code or architecture description.
- The parent candidate and lineage.
- The model and configuration used to generate it.
- The problem specification.
- The evaluation environment and version.
- Test results, costs, and failure information.
- Reasons for accepting or rejecting the candidate.
The archive should preserve useful diversity, not simply keep a copy of the current winner.
A low-performing candidate may have introduced a useful idea that becomes valuable in a different context.
The Darwin Gödel Machine illustrates why this matters. Its archive permits exploration through multiple lineages, including lineages that temporarily perform worse before producing a more successful descendant.
- A sandboxed execution environment
Candidate systems must be tested somewhere they cannot accidentally damage the production environment.
The sandbox should have explicit resource limits, restricted permissions, controlled dependencies, and a clear policy for network access.
A candidate should not inherit access to production credentials just because its parent happens to have them.
This is especially important when AI-generated code is executed rather than merely inspected.
- An evaluator
The evaluator executes predefined tests and measures candidate performance.
Depending on the task, it might combine deterministic unit tests, integration tests, benchmark scores, randomized trials, simulated environments, or external validation.
Not every evaluation needs an LLM.
For questions with objectively measurable answers, deterministic tests are usually preferable.
For questions that need semantic judgment, a model-based evaluator may help, but it should not be treated as an unquestionable authority.
- An admission gate
Passing an evaluation does not automatically grant production access.
A gate decides whether the candidate has met the requirements for the next stage.
The gate should be separate from the candidate being evaluated.
A candidate should not be allowed to bypass the gate by changing its source code, rewriting its acceptance criteria, or modifying the tests that decide whether it passes.
- A feedback mechanism
The system needs to turn successful and unsuccessful experiments into information useful for the next round.
That might mean storing which strategies worked on which task families, identifying repeated failure patterns, or adjusting how new candidates are generated.
This is where the archive becomes more than a collection of programs.
It becomes the history of an evolving search process.
But the history should not become a substitute for testing. A candidate that worked yesterday may fail under a different environment, dependency version, or task distribution.
- A reference implementation in pseudocode
The following is a conceptual sketch, not a runnable implementation. The proposal engine, sandbox, evaluator, and archive would each need their own implementation and tests.
def evolve_systems(seed, task_suite, budget, archive, evaluator):
archive.record(seed)
while budget.remaining(): parent = archive.select_parent( preserve_diversity=True ) candidate = propose_new_system( parent=parent, task_suite=task_suite, failure_history=archive.failure_history() ) if not validate_candidate_structure(candidate): archive.reject(candidate, reason="invalid_structure") continue if not static_security_checks(candidate): archive.reject(candidate, reason="security_gate") continue result = evaluator.run_in_sandbox( candidate=candidate, tasks=task_suite.public_tasks, fixed_budget=budget.per_candidate ) if not result.execution_valid: archive.reject(candidate, reason="execution_failure") continue independent_grade = evaluator.grade_held_out( candidate=candidate, tasks=task_suite.sealed_tasks, fixed_budget=budget.per_candidate ) decision = evaluate_admission( baseline=archive.baseline, candidate=candidate, public_result=result, private_grade=independent_grade, hard_constraints=task_suite.hard_constraints ) archive.record( candidate=candidate, public_result=result, private_grade=independent_grade, decision=decision ) if decision.accepted: archive.add(candidate) return archive.best_verified() The interesting line is not "propose_new_system".
It is the admission decision.
The proposal engine can be creative, surprising, and occasionally wrong.
The evaluator needs to be much less impressed by confidence.
A production implementation would also need to prevent candidates from inspecting sealed test cases, control random seeds where possible, record dependency versions, handle evaluator failures, enforce actual compute and financial budgets, and preserve the ability to roll back an admitted candidate.
The example leaves those details abstract on purpose. A trustworthy research system needs more than a short Python loop.
- Why the archive should not select only the current best candidate
Greedy optimization is tempting.
You have ten candidates. Candidate seven has the highest score. Obviously, create the next candidate from candidate seven.
Except that this can eliminate alternative approaches too early.
Imagine searching for a better routing algorithm.
One lineage makes small improvements to a familiar heuristic. It performs well on the current benchmark.
Another lineage initially performs worse because it explores a different search strategy.
If you delete the second lineage immediately, you may never discover the combination of ideas that eventually outperforms the first.
This is not an argument for keeping every failed candidate forever.
Storage and compute are limited.
It is an argument for keeping enough strategically different candidates to avoid collapsing the search into one locally successful path.
The archive might use performance, novelty, diversity, task coverage, and the historical usefulness of a lineage to decide what to explore next.
A simple approach would preserve several high-performing candidates alongside candidates that represent genuinely different strategies.
A more advanced approach could allocate exploration based on uncertainty or expected information gain.
The exact selection method would depend on the task and the available budget.
The general principle is straightforward:
The next candidate should sometimes come from the best solution and sometimes from an alternative that might teach the search process something different.
This is one reason "keep making the best version slightly better" and "search for new architectures" should not be treated as the same operation.
- The evaluator is the most important part of the loop
Let's consider what happens when a system uses benchmark scores to select its own descendants.
It generates a candidate.
It runs the candidate.
It observes a score.
It keeps the candidate if the score is higher.
The problem is that a better score may have several explanations.
The candidate may have discovered a genuinely better algorithm.
It may have found a shortcut that works only on the benchmark.
It may exploit an implementation bug in the evaluator.
It may overfit to the evaluation set.
It may simply consume more resources.
The score alone does not tell us which explanation is correct.
In fact, self-improvement makes this problem more important because every accepted modification changes the system that proposes future modifications.
A weakness in the evaluation process can become a selection pressure across the entire lineage.
If cheating improves the score, the search process can gradually produce systems that are better at cheating.
That is why I would treat independent evaluation as part of the core architecture rather than a final quality check.
A useful design has at least three boundaries.
The candidate can see public feedback. It needs enough information to improve.
The admission process can use private evaluation data. The details of those tests should remain hidden from the candidate.
The production gate applies constraints that a candidate cannot remove by modifying its own code.
Even then, private tests are not magic.
An agent that receives repeated aggregate scores can still adapt to them. An imperfect evaluator can still reward undesirable behavior. Test data may overlap with the training data of an underlying foundation model.
The aim is not to claim that an evaluator can never be fooled.
The aim is to make cheating harder to discover, measure its effects, and prevent one candidate from unilaterally redefining success.
- A concrete warning from AI Scientist research
A useful critique is the preprint "The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems" (https://arxiv.org/abs/2509.08713).
The researchers examined potential problems in automated scientific workflows, including benchmark selection, data leakage, metric misuse, and post-hoc selection bias.
Their controlled experiments with two open-source AI scientist systems found that some generated research workflows made undocumented dataset choices, including creating synthetic datasets or subsampling existing datasets. They also found that internal reward mechanisms could favor experiments with strong test performance, creating a risk of selecting favorable results after the fact.
The paper did not find evidence of deliberate test-set inspection in those systems. The concern was subtler: test information could still influence selection through the internal reward mechanism.
This distinction matters.
A failure does not have to be intentional to corrupt an evaluation.
A system can follow its coded procedure and still produce a misleading result because the procedure rewards the wrong behavior.
Another critical discussion, "AI Scientists Fail Without Strong Implementation Capability" (https://arxiv.org/abs/2506.01372), reviewed 28 research papers generated by five AI scientist systems and argued that implementation and rigorous verification remain important bottlenecks.
These works do not prove that automated scientific discovery is impossible. They show why generating convincing research output is not enough.
For a Third Layer, this is directly relevant.
If the system creates new agents, and those agents create further agents, the selection process becomes responsible for determining which ideas propagate.
The evolutionary process can amplify either useful discoveries or systematic mistakes.
That is why the evaluator, the audit trail, and the admission gate must be treated as first-class architectural components.
Part V: How do we prove the system has done something genuinely new?
- A successful benchmark run is not enough
Suppose a system creates a candidate agent that improves its benchmark score from 70% to 80%.
What have we proved?
We have evidence that the candidate performs better on the measured benchmark under the tested conditions.
That is valuable.
But it does not automatically establish that the candidate is more capable in general, that its architecture is genuinely novel, or that the discovery process will continue improving.
We need to separate several claims.
Claim| Evidence required
The candidate runs correctly| Execution tests, error handling, reproducible results
The candidate improves its target task| Comparison with a baseline under equivalent conditions
The improvement generalizes| Held-out tasks or environments not used for selection
The system discovers diverse architectures| Multiple meaningfully different candidates with documented lineage
The system creates useful specialists| Evidence that different descendants outperform alternatives on distinct task families
The process improves recursively| A discovered agent improves subsequent generations more effectively than the previous method
Progress accelerates| Repeated, controlled evidence that improvement per fixed budget grows across generations
These claims require different experiments.
A system can satisfy some and fail others.
For example, a population of agents may contain several useful specialists without demonstrating accelerating improvement.
A recursive optimizer may produce better generations while operating on only one narrow class of system.
We should report those outcomes separately rather than compressing all of them into one headline about self-improving AI.
- A test protocol I would actually trust
If I were evaluating a candidate Third Layer architecture, I would begin with a controlled experiment.
Not with a press release.
Not with a chart that has a very optimistic upward slope.
With a protocol that another team can inspect and reproduce.
Step 1: Establish a baseline
Choose an initial system and a task suite.
Measure its success rate, cost per successful task, resource consumption, failure categories, and security regressions.
Record the exact software versions, model versions, dependency lockfiles, and benchmark identifiers.
The baseline must be strong enough to make the comparison meaningful.
Step 2: Define the acceptance criteria before the experiment
Decide what counts as improvement before the candidates are generated.
For example, a new candidate might need to:
- Improve task success under a fixed compute budget.
- Pass all required safety and correctness tests.
- Avoid unacceptable regression on protected tasks.
- Generalize to a...
