Miadi Review

Agentic Systems Engineering and Continuous Evals with Braintrust

Aug 27, 2026 · model: manual-edit · channel: How I AI

Watch source

Agentic Systems Engineering and Continuous Evals with Braintrust

Source: Braintrust CEO: Evals are the new PRD for AI products — How I AI https://youtu.be/QE_1hRLsehM

Potential install

curl -fsSL 'https://www.braintrust.dev/wizard/setup.sh' | sh -s -- --from in_app_onboarding --app-url 'https://www.braintrust.dev' --api-url 'https://api.braintrust.dev' --org-id '35b4ca8b-e91c-4c29-b3c3-6092340878eb' --proj-id '24e8c16a-31d5-4031-9365-b4e5be659b78'

  • I dont know what it does though...

Summary

In this episode of How I AI, Braintrust CEO Ankur Goyal and host Claire Vo explore how agentic coding and systematic evaluation transform software engineering. Autonomous coding agents tackle complex systems engineering, such as database optimization, by running continuous benchmarks in parallel. Goyal shows how establishing rigorous evals shifts engineering from specifying implementation details to defining objective quality constraints. By codifying expert taste into automated scorers and leveraging robust CI pipelines, teams automate architectural exploration and maintain uncompromising quality standards.

Aspects

Autonomous Architectural Exploration and Benchmarking (04:21-05:58)

Engineers utilize autonomous coding agents to run dozens of rigorous benchmarks across database formats and execution engines. Instead of hand-writing isolated tests, agents systematically explore architectural tradeoffs, such as column-store formats and bloom filter indexing, directly on near-production datasets [04:21-05:58].

The Agent Line and Protecting Maker Time (14:16-15:27)

Engineering workflows shift by moving execution below the 'agent line.' By running multiple concurrent agent sessions in tmux and delegating routine implementation, leaders and engineers protect uninterrupted afternoon maker schedules to focus on deep technical problems and architectural direction [14:16-15:27].

Evals as Executable PRDs Defining the 'What' (23:48-26:00)

Machine learning and LLM engineering redefine product requirements: specifying what success looks like rather than dictating how code executes. Teams define objective evaluation criteria and dataset inputs, allowing models and agents to explore optimal solutions without falling into the 'whack-a-mole' trap of manual spot-checks [23:48-26:00].

Codifying Human Taste into Automated Rubrics (30:01-33:13)

Qualitative feedback from expert taste-makers is codified into formal eval scorers. When human review uncovers flaws during periodic vibe checks, that critique is converted into automated scoring rules, scaling high standards across the codebase without creating human review bottlenecks [30:01-33:13].

Carving Complexity and Investing in CI Velocity (34:25-36:30)

Agent-assisted development accelerates feature generation, shifting engineering's focus toward carving away unnecessary complexity. To sustain high agent throughput and prevent regressions, teams prioritize heavy investment in CI pipelines, treating automated verification infrastructure as the foundational enabler of development velocity [34:25-36:30].

End-results and Core Aspect

Core Aspect — Closed-Loop Agentic Infrastructure Optimization and Continuous Evaluation (09:44-11:32)

Software engineering transitions from manual code construction to defining precise evaluation criteria and managing parallel agent workflows. Agents autonomously explore complex architectural spaces, benchmark system performance on production-like data, and iterate against automated eval suites until performance thresholds and domain-expert standards are met [09:44-11:32].

Results

  • Autonomous Column-Store and Index Optimization (09:44-10:05): Parallel remote agents conducted week-long autonomous benchmark experiments across various column-store formats and execution engines, discovering an optimal bloom filter indexing strategy for high-concurrency multi-billion trace querying [09:44-10:05].
  • Automated Eval Scorer Synthesis in Playground (27:14-30:20): Using natural language constraints, engineers generated multi-criteria LLM evaluation scorers inside Braintrust Playground to grade documentation Q&A responses, automatically assessing conciseness, language restrictions, and formatting compliance across test datasets [27:14-30:20].
  • Codified Expert Taste Multiplying Quality Bar (30:01-33:13): Periodic qualitative reviews from key designers were converted into programmatic eval functions, allowing the team to scale one designer's discerning product standard across dozens of autonomous agent workflows without introducing review delays [30:01-33:13].