Miadi Review

Loop-to-Graph Analysis: Andrej Karpathy's Autoresearch Architecture and Use Cases

Oct 8, 2026 · model: manual-import · channel: Greg Isenberg

Watch sourceView transcript

In the chronicle

Loop-to-Graph Analysis: Andrej Karpathy's Autoresearch Architecture and Use Cases

Source: Karpathy's "autoresearch" broke the internet — Greg Isenberg https://youtu.be/qb90PPbAWz4

Intent

self-evolve miadi-factory

Summary

Greg Isenberg presents Andrej Karpathy's autoresearch repository (github.com/karpathy/autoresearch, March 2026) as a loop that runs without a person. The agent edits one training file, runs a five-minute training on one GPU, compares the validation score with the best so far, and keeps the commit only when the score improves. The presenter proposes the same loop for landing pages, trading backtests and finance operations. These are business ideas, not results shown in the video.

Academic Fields

Hyperparameter Optimization and AutoML (Machine Learning)

Studies how to choose a model's settings by automated search instead of by hand. Autoresearch is one, with an agent proposing each trial under a fixed five-minute budget. The field's limit: the best of many trials on one validation set looks better than it is, so a gain needs a held-out check.

Search-Based Software Engineering and Genetic Improvement (Software Engineering)

Treats changing a program as search: generate variants, score them with a fitness function, keep the fittest. Autoresearch does this with an LLM writing the variants of train.py, as FunSearch and AlphaEvolve do. The field's limit: the fitness function defines what counts as better, and a patch can pass every test yet be wrong.

Autonomic Computing and the MAPE-K Loop (Systems Engineering)

Designs systems that monitor themselves, analyze, plan and execute changes over shared knowledge. The overnight loop is one such cycle, and its results.tsv log is the knowledge. The field's limit: the loop carries out goals and policies that people supply. It has no test of whether the goal itself is the right one.

Self-Improving Language-Model Agents (Artificial Intelligence)

Studies agents that revise their own prompts, memory, skills or code from feedback, as in Reflexion, Voyager and the Darwin Gödel Machine. The video's claim that the loop fits any codebase sits here. The field's limit: an agent can satisfy its evaluator without doing the task, so the evaluator needs its own audit.

Double-Loop and Organizational Learning (Organization Studies)

Distinguishes correcting actions against a fixed rule (single loop) from questioning the rule itself (double loop). Autoresearch is single loop: val_bpb never changes. The person editing program.md is where double-loop learning can happen. The field's limit: Argyris found that defensive routines block double-loop learning, so it rarely happens without being asked for.

Metric Validity: Goodhart's Law and Overfitting (Statistics and AI Safety)

Studies what happens to a measure once it is optimized. The video's landing-page tests and trading backtests sit here. The field's limit: pressure on a proxy degrades it, and many trials against one backtest or test set produce gains that are partly selection noise.

Aspects

The Autonomous Optimization Loop (00:46-01:46)

An LLM agent plans a change, edits train.py, runs a five-minute training on an NVIDIA GPU, and reads val_bpb (validation bits per byte, lower is better). It keeps the commit when the score improves, else resets. The person writes program.md, the instructions. The agent does not edit it. The loop runs unattended overnight [00:46-01:46].

Generalizing Beyond Model Training (02:08-02:29)

The video quotes Shopify's CEO, Tobi Lütke: autoresearch "works even better for optimizing any piece of software" with an auto folder and a program.md. Any codebase with instructions and a script that scores a run can host the loop [02:08-02:29]. The video shows no measured case outside model training.

Human-Out-of-the-Loop Operational Model (03:48-04:40)

The person writes a clear task and gives the agent the code and a GPU. The agent runs code, reads the results, updates its plan and logs metrics. The person returns hours later to a written summary and the runs that were kept [03:48-04:40]. Review happens after the fact, not during the loop.

AgentHub and Multi-Agent DAG Collaboration (18:28-19:27)

AgentHub, Karpathy's next project, is described as GitHub for agents: a bare git repository and a message board, with no main branch, no pull requests and no merges. Agents push to a directed acyclic graph of commits and coordinate on the board [18:28-19:27]. On 2026-10-08 karpathy/agenthub answered 404. The fork ottogin/agenthub preserves it.

End-results and Core Aspect

Core Aspect — Autonomous Benchmark-Driven Code Synthesis (01:03-02:05)

The loop is closed by one fixed metric. The agent changes code, measures the run against val_bpb, and keeps only the changes that improve it, without a person in the loop [01:03-02:05]. The person stays outside it, writing the instructions and choosing the metric.

Results

The first item is the repository's own use. The other three are ideas the presenter proposes for selling the loop. The video shows no measurements for them.

  • Automated ML Hyperparameter Exploration (00:04-00:09, 21:54-22:20): The agent runs one five-minute training after another and keeps the changes that lower val_bpb. The repository is tested on one H100. The presenter ran it on a Google Colab T4 GPU [00:04-00:09, 21:54-22:20].
  • Continuous Marketing Funnel Mutation (06:49-07:49): Proposed: the agent writes variants of headlines, layouts and offers, sends traffic to them, and keeps the one that converts better, as A/B testing tools such as Optimizely do [06:49-07:49].
  • High-Volume Quantitative Backtesting (12:06-12:46): Proposed: overnight backtests of many simple trading rules on one GPU, keeping the few that look promising [12:06-12:46]. Many trials against one backtest overfit (see Academic Fields).
  • Back-Office Exception Elimination (14:22-15:04): Proposed: the loop improves the rules and prompts for invoice matching, expense reports and exception detection. "We cut your AP, expense time in half" is a sales line the presenter suggests, not a measured result [14:22-15:04].

Related Review

Engineering Grounded Agent Resumption and Provenance in Miadi: its v17 Future Discussion maps this loop onto Miadi. Resumption records are what the loop reads, Episode 351's tasks are its score, and each kept change carries the custody that review asks of every claim. Both are discussed in ⟐ ep 551 Learning Between Sessions: Autoresearch for the Miadi Factory.

Sources