---
title: "Deterministic Rails for Non-Deterministic AI"
event: "PhRUG (Philippine Ruby Users Group)"
date: 2026-10-27
duration: 40
description: "Early ideas from a master's capstone on Rails project setup and coding-agent workflows. Published benchmarks and concrete checks open a discussion about what helps agents deliver better changes, what it costs, and how to measure it."
slug: "deterministic-rails-for-non-deterministic-ai"
---
PhRUG Meetup • October 2026 • Narra Labs, Quezon City
# Deterministic Rails for Non-Deterministic AI
### Quality Engineering & Harness Design for Coding Agents
Speaker: Artem Blazhievskii • Community: Philippine Ruby Users Group (PhRUG)
Early ideas and examples. An invitation to compare notes.
Note:
Mabuhay, PhRUG! I'm Artem. Today I want to explore a practical question with you: when an agent says the feature is done, what evidence should we ask for?
I'm still early in figuring this out. Working with agents often makes me feel like a junior again, and I suspect some of you know that feeling. I'm sharing ideas to try, and I'd like to hear what works differently in your projects.
We'll look at published Agents on Rails results, try a small Rails example, and discuss a possible workflow. The benchmark tells us how those runs scored. Whether my proposed combination of checks improves delivery is a question I still need to test.
*Pause, make eye contact, then move to the introduction.*
**Rehearsal plan, not measured delivery times:** introduction, Rails toolkit and capstone context by 6:00; benchmark discussion by 10:30; workflow and architecture by 12:30; schema/static checks and Herb demo by 21:00; runtime checks by 27:30; verification workflow by 33:00; takeaways by 35:00. Reserve 35:00–40:00 for questions. Validate these checkpoints in rehearsal. Technical references and sources below the spoken notes are backup for questions, not part of the spoken script.
---
FULL-STACKRUBYONRAILSDEVELOPER
ARTEM BLAZHIEVSKII
I'm Artem (@ablzh), a full-stack Rails developer and creator of Ruby Overdrive. I build and run Pasabaya.app and CSMasters.dev, from database design to the interface and deployment.
Ruby community: I translate ruby-lang.org into Russian and help keep its Russian content up to date.
Hey, it's me! Yep, my name is Artem pronounced pretty much like Artemis (the ancient Greek goddess of the hunt)
Note:
I'm Artem, a full-stack Rails developer, and I write Ruby Overdrive. I build and run my own projects, including Pasabaya and CSMasters. I'll show you what they do in a moment.
I also try to contribute to the Ruby community. I translate ruby-lang.org into Russian and help keep the existing Russian pages up to date as the original content changes. Translation is one way I can make Ruby more accessible to other developers.
*Point to the community line. Transition: “One reason I use Rails for my projects is the idea of the one-person framework.”*
Sources: [Russian Ruby website](https://www.ruby-lang.org/ru/); [October news translations, merged PR #4052](https://github.com/ruby/www.ruby-lang.org/pull/4052); [updates to existing Russian translations, PR #4030](https://github.com/ruby/www.ruby-lang.org/pull/4030). PR #4030 is open as of October 29, 2026. This describes translation contributions and ongoing updates, not an official maintainer role.
---
Why Rails
## The One Person Framework
Rails and its ecosystem: a toolkit for building the whole product.
| Product layer | Tools that work together |
|---------------|--------------------------|
| **Backend** | Active Record, Action Controller, Action Mailer, Active Storage |
| **Frontend** | Action View + Hotwire: **Turbo & Stimulus** |
| **Solid Trifecta** | **Solid Queue** for jobs, **Solid Cache** for caching, **Solid Cable** for pub/sub |
| **Mobile apps** | **Hotwire Native**: separate iOS & Android clients reusing web screens |
| **Deployment** | **Kamal**: deploy application containers to your servers |
Why this appeals to me: One developer can work across the product, with shared conventions and tools for each layer.
Idea: DHH, The One Person Framework. The toolkit shown here reflects Rails 8 and its ecosystem.
Note:
DHH calls this the one-person framework. I read that as a design goal: make building a complete product approachable for an individual developer.
*Move down the table, connecting each name to the job it does.*
Rails covers the backend and server-rendered views. Turbo and Stimulus add frontend interactions. The Solid Trifecta is Solid Queue for background jobs, Solid Cache for caching, and Solid Cable for passing messages between Action Cable processes, using the database.
Hotwire Native extends the web application into separate iOS and Android apps, reusing web screens with native navigation. Kamal handles deploying the server application in containers.
For me, the appeal is having these pieces designed to work together. I can follow a feature from the database to the interface and production. Let me show you two projects where I use this approach.
**Technical reference:** DHH introduced the phrase in the linked 2021 Rails 7 essay. This slide maps the idea to the Rails 8 ecosystem; it is not a claim that the 2021 release included every tool listed. Rails 8 applications are generated with Hotwire, Solid adapters and Kamal configuration by default, subject to generator options and environment configuration. Solid Cable is the database-backed pub/sub adapter for Action Cable, not the complete WebSocket implementation. Hotwire Native is a companion framework: iOS and Android projects still need native setup, builds and distribution; native features may require Swift or Kotlin. Kamal deploys the server application, not the mobile binaries, and requires server access and deployment configuration. No delivery-speed or staffing ratio is claimed here.
Sources: [DHH's original essay](https://world.hey.com/dhh/the-one-person-framework-711e6318); [Rails 8 defaults and Solid adapters](https://rubyonrails.org/2024/11/7/rails-8-no-paas-required); [Hotwire Native architecture](https://native.hotwired.dev/overview/how-it-works); [native screens](https://native.hotwired.dev/overview/native-screens); [Kamal](https://kamal-deploy.org/).
---
## Pasabaya.app
Community carpooling in the Philippines
Search by route and dateTrip details, available seats and seat requestsPrivate trip chat: example conversation from the homepage
Note:
Pasabaya is my open-source carpooling project for the Philippines. It helps people find someone already heading their way.
*Point to the route and date fields. Advance once to show the trip details.*
Passengers choose a departure and request a seat. Regular rides have separate seat availability for each departure.
*Advance again to the chat example from the homepage.*
Once the driver accepts a request, accepted passengers can coordinate pickup in a private trip chat. This example shows the driver suggesting a meeting point and the passenger confirming. The chat stays tied to the trip and its participants.
I build the Rails application and its Hotwire interface, and deploy it with Kamal. This is the kind of work I mean by full-stack: the database rules and the screens have to agree. The next project comes from my experience teaching Computer Science.
*Keep the focus on the three user flows; save implementation details for questions.*
Sources: [Pasabaya](https://pasabaya.app/); [source code](https://github.com/ablzh/pasabaya). Screenshots captured from the public [route search](https://pasabaya.app/rides/from-makati-to-quezon-city), [trip details](https://pasabaya.app/rides/111-makati-to-quezon-city), and [homepage chat example](https://pasabaya.app/#home-chat) on October 29, 2026. The second image shows the signed-out seat-request entry point. The third is the homepage's illustrative conversation, not a private user conversation. Displayed dates and availability are a snapshot, not usage metrics.
---
## CSMasters.dev
Interactive Computer Science learning for Cambridge students
Run pseudocode and follow variable values through the execution
AI Tutor, inside the lesson
A question about binary overflow:
Why does 200 + 80 overflow an unsigned 8-bit register?
The tutor recalls the limit of 255, then asks what happens to the extra bit.
Real demo conversation · AI integration built with RubyLLM
Note:
I built CSMasters after teaching Cambridge Computer Science. Students needed somewhere to run the pseudocode they were learning and understand what happened to their variables.
*Point to the code on the left, then the trace table on the right.*
The sandbox executes the program and records the variable values at each step. Students can compare the trace with what they expected. The platform also has interactive lessons with feedback.
*Advance once to show the AI Tutor. Point to the question, then the reply.*
I also built an in-lesson AI tutor with RubyLLM. It uses the lesson and selected exercise as context. Here I asked why 200 plus 80 overflows an unsigned 8-bit register. The tutor reminds me that the maximum is 255, then asks how many bits the full result needs and what happens to the extra bit. The aim is to guide the learner's reasoning with questions and hints. That's the model's instruction, not a guarantee that every answer will be correct.
These projects are why I care about today's topic. A feature has to work from the database through to the interface, and I want evidence I can inspect when an agent changes it. That question also connects to my master's capstone project.
*Pause long enough to see the code, trace and Tutor reply. Move to the capstone context next.*
Sources: [CSMasters](https://csmasters.dev/); [pseudocode sandbox](https://csmasters.dev/compiler); [lesson used for the Tutor demo](https://csmasters.dev/learn/igcse-0478/binary-addition-and-overflow?course_module_slug=1-data-representation). Both screenshots were captured on October 29, 2026. The first shows the sandbox's starter program with the side-by-side variable trace; its execution time is not a performance benchmark. The second shows a real Tutor response to a demonstration question in the speaker's administrator account, using the account's test AI access. It is not a student conversation or evidence of learning outcomes. The screenshot shows the narrow-screen layout, cropped to the Tutor panel; the adjacent text paraphrases the visible exchange. RubyLLM and lesson/task context were also checked in the project's implementation. CSMasters is an independent educational platform. These project examples establish the speaker's work; they do not claim that the proposed quality pipeline has been evaluated on either project.
---
Work in Progress
## My Master's Capstone
How can project setup, executable checks and a clear workflow help coding agents deliver better changes?
For me, “better” needs evidence:
- Do changes meet the agreed requirements?
- Which defects reach review, and what is the total time and cost?
Early steps: I'm developing the approach. Measured conclusions are still ahead, and I'd value your experience.
Note:
I've chosen this area for my master's capstone project. I'm grateful to my adviser for being open to a topic that is still developing and supporting me in exploring it.
*Point to the working question. Keep the tone personal and conversational.*
I want to investigate how to prepare a project, surround it with useful checks, and build a workflow that helps coding agents. By “better,” I mean things we can examine: whether the change meets the requirements, which defects still reach review, how much human rework is needed, and the total time and cost of implementation and verification.
I'll call the setup around the agent a harness: its tools and instructions, plus the code that runs checks, collects results and controls the repair loop.
I'm taking the first steps. Over time, I want to compare approaches under the same tasks and model settings and draw conclusions supported by data. Today's ideas are part of that exploration. I'd be glad to hear about checks that helped you, checks that got in the way, and results that surprised you.
*Transition: “One place to start is the cost of making the next change.”*
**Reference:** This is a working research question, not the project's formal title or a completed study. The capstone context comes from the speaker. The proposed comparisons and measures describe future work; this talk reports no capstone results. **Rehearsal checkpoint:** finish the introduction around 6:00.
---
Act I: The Economics
## Rails vs Rust: The Cost of Change
After DHH's HEY/Rust discussion: what does it cost to add the next feature?
**Campfire Bots:** the same task and checks, Rails versus a specific Rust port.
| Model | Agent cost: Rails / Rust | Agent time: Rails / Rust |
|-------|-------------------------|-------------------------|
| **Claude Opus 5.5** | **$5.67 / $20.52** | **35 / 85 min** |
| **GPT-6 Astra** | **$7.14 / $14.17** | **27 / 59 min** |
Before a rewrite: Weigh runtime savings against the cost of changing the product. This experiment measured agent work, not long-term maintenance costs.
Source: Evil Martians, October 21, 2026. The AI-generated Rust port contains roughly 13x more code.
Note:
DHH's HEY and Rust discussion raises an interesting question: what happens to the cost of the next feature after a rewrite?
*Point to the Opus row, then the Astra row.*
In this Campfire experiment, both models completed both versions with similar scores. The Rust version took more agent time and spending. These columns are agent costs and elapsed agent time. They don't measure application response time or developer salaries.
There is an important detail about the comparison: this was a particular Rails-shaped Rust port that rebuilt framework infrastructure. Its maintenance and operating costs need their own measurements. For our purposes, the useful question is the total cost of changing the product.
*Pause.* Even with Rails' conventions, agents still miss requirements. Let's look at those results.
**Reference:** This experiment does not establish ratios for idiomatic Rust applications or the total economics of a rewrite. HEY runtime figures are outside the evidence presented here.
Sources: [Evil Martians comparison and caveats](https://evilmartians.com/chronicles/the-agents-on-rails-files-designing-black-box-evals-for-coding-agents); [DHH's Rails World opening keynote](https://www.youtube.com/watch?v=vDjW_dRyKXY).
---
Act I: The Reality Check
## Feature Tickets: What the Benchmark Measures
**Agents on Rails:** mean task scores with partial credit, October 15 data snapshot.
| Model (all at xhigh effort) | Stage 1: Writebook | Stage 2: Fizzy |
|----------------------------|-------------------|----------------|
| **Claude Opus 5.5** | 69.2% | **59.2%** |
| **GPT-6 Astra** | 70.8% | **57.8%** |
| **GPT-6.1 Sol** | 72.4% | **52.9%** |
Stage 1: **19 tasks**. Stage 2: **20 tasks**. **Three runs per task and configuration.**
Read the metric correctly: 59.2% is a mean score, not the share of fully solved tickets. These results do not establish a universal ceiling.
Sources: Rails Foundation / Evil Martians, fixed dataset.
Note:
Before reading the scores, let's agree on what they mean.
*Point to 59.2% in the Fizzy column. Pause.*
That is a mean task score with partial credit. It isn't the percentage of tickets fully solved. The site averages the runs for each task, then averages those task scores.
Required checks determine whether a run earns any credit. Optional checks add weighted credit, and Stage 1 also rewards idiomatic API use. Writebook and Fizzy have different tasks and grading rules, so the difference between columns doesn't isolate the effect of task size.
The benchmark already uses an agent harness and hidden deterministic checks. The workflow I'll propose today has not been evaluated against that hidden suite. Our next question is which requirements we can make explicit in our own repositories.
**Technical reference:** Stage 2 has 60 runs per configuration. The benchmark uses lemans and miniswen. The September binary results, Astra medium 21/60 and GPT-5.6 Luna 0/60, belong to an earlier experiment; they are not October fractional scores. GPT-5.6 Luna and GPT-6 Luna are different model names. These results describe the stated tasks, settings, and rubric, not a universal model ceiling.
Sources: [October grading changes](https://rubyonrails.org/2026/10/13/agents-on-rails-stage-3), [September Stage 2 report](https://rubyonrails.org/2026/9/9/agents-on-rails-stage-2), [dataset used for this slide](https://github.com/rails/website/blob/1d3c37587b35ed90df92d9bbaa41a5a9227edcef/_data/ai_evals.json).
---
Act I: Failure Modes
## Five Failure Modes to Check
Benchmark observations motivate checks for these Rails risks:
1. Migration mistakes: Handwritten migrations need checks for ordering, DDL safety, rollback behavior, and data preservation.
2. Broken interactions: An HTTP 200 check can pass while a Turbo Frame or Stimulus interaction fails.
3. Framework bypass: Custom logic can overlook existing APIs such as delegated_type or params.expect.
4. Runtime concerns: Job retries, persistent queues, authorization, and N+1 queries need explicit checks.
5. Weakened tests: A green suite can hide a changed assertion and a broken requirement.
Note:
Here are five places where I would ask for evidence: migrations, browser interactions, framework APIs, runtime behavior, and the tests themselves.
*Reveal the bullets in order. Pause briefly on migrations and weakened tests.*
The published work discusses migration generation, API choices, background jobs, and changed tests. I'll use small Invoice and query examples to make the checks concrete. Those snippets are teaching examples, rather than code taken from the benchmark runs.
The useful move is to turn a requirement into a check the agent can actually run. Let's start with the instructions we already put in our repositories.
**Reference:** These five categories have no measured frequency attached. The Rails at Scale slides report generator usage in 27 of 2,460 runs; that is not a migration or rollback failure rate. **Rehearsal checkpoint:** finish the benchmark block around 10:30.
Sources: [Nazarova and Dementyev, Rails at Scale, slides 16–19](https://speakerdeck.com/irinanazarova/agents-on-rails-rails-at-scale-2026), [Stage 3 observations on jobs and test changes](https://rubyonrails.org/2026/10/13/agents-on-rails-stage-3).
---
Act I: Instructions & Feedback
## Which Instructions Can Become Checks?
```markdown
# Example project instructions
Please remember:
- Never write N+1 queries!
- Always use deliver_later, never deliver_now!
- Always make background jobs idempotent!
- Ensure your migrations are reversible!
- Use our design system, no custom CSS!
```
The Gap: Instructions express intent. Executable checks provide evidence that a change satisfies specific requirements.
I'm exploring how repository instructions and executable checks can work together.
Note:
These are reasonable things to ask for. The question is how the agent knows whether it has followed them.
*Point to “Never write N+1 queries,” then “Ensure your migrations are reversible.”*
A repository instruction expresses our intent. An executable check gives feedback on a specific result. We can keep the useful instructions and connect them to commands: run this check, inspect the failure, and fix the code.
Some requirements need a scenario test or a human decision. The next step is to agree on those acceptance criteria before the agent starts a substantial change.
**Reference:** The evidence for this talk contains no measured 500-line or 50k-token failure threshold. The quality workflow is an engineering recommendation, not a quotation or benchmark finding.
---
Act I: A Workflow to Explore
## One Workflow to Explore
A workflow for non-trivial changes, from Kirill Mokevnin's *Agentic Coding Design Patterns*:
01
Explore
Read the code, schema and tests.
02
Plan
Agree on the outcome and acceptance checks.
03
Code
Build a small slice across UI, API and DB.
04
Commit
Verify the change. Review the evidence.
A starting point: For substantial changes, I'd agree on acceptance checks before implementation and keep the planning proportional to the task.
Note:
Kirill Mokevnin describes four phases for non-trivial work: Explore, Plan, Code, and Commit.
*Follow the phases from left to right, stopping at Plan.*
Here we agree on the outcome, the boundaries, and the checks. A vertical ticket should produce something usable across the UI, application, and database. Then the agent has a concrete target, and the reviewer has something concrete to accept.
For a simple mechanical edit, the process can be much smaller. The amount of planning should match the change.
*Point to the verification step.* Now let's put the checks around that implementation phase.
**Reference:** Matt Pocock's grilling and specification skills are optional aids. Their names and invocation depend on the installed skill set; the workflow requires acceptance criteria, not a specific slash command.
Sources: [Four Phases](https://github.com/mokevnin/agentic-coding-design-patterns/blob/main/book/en/explore-plan-code-commit.md), [Matt Pocock's skills](https://github.com/mattpocock/skills).
---
Act II: Architecture
## Four Layers of Project Checks
The agent writes a change. Each layer checks a different property.
1. Schema contextAnnotateRb keeps schema metadata close to the model.
3. Runtime scenariosExercise SQL, migrations and browser interactions.
4. Verification loopReturn diagnostics, limit repairs and collect evidence.
“Deterministic” here: explicit pass/fail rules with controlled inputs. The combined effect still needs measurement.
Note:
These four layers are one model I'd like to explore in my capstone. I want to find out which checks help agents and what they cost to maintain.
By “deterministic,” I mean explicit rules whose pass/fail conditions we can inspect. Runtime checks also depend on data, clocks and services, so reproducible execution takes setup.
*Trace the four layers from top to bottom.*
First, give the agent current schema context. Then run static checks while feedback is cheap. Next, exercise the database and browser behavior. Finally, collect the results for the exact change under review.
The agent can repair a failed check, but the loop needs an attempt or time limit. An unresolved failure comes back to a person. We'll measure each stage on the actual project before giving it a time budget.
Let's start with the information already available in a Rails database.
**Technical reference:** Static checks, database checks, and browser checks require different environments. Record the tested source state, tool versions, commands, and results. The complete pipeline is a proposal; this talk has no measured implementation of it. **Rehearsal checkpoint:** reach schema and code examples around 12:30.
---
Layer 1: Grounding
## AnnotateRb: Schema Context
**AnnotateRb** puts schema metadata beside the model. Shortened illustrative example:
```ruby
# == Schema Information
# Table name: invoices
# id :bigint not null, primary key
# amount_cents :integer default(0), not null
# status :string default("draft"), not null
# customer_id :bigint not null
class Invoice < ApplicationRecord
end
```
Check against a prepared verification database:
```bash
bundle exec annotaterb models --frozen
bundle exec annotaterb routes --frozen
```
Freshness check:--frozen fails when annotations would change. It does not validate column access in application code.
Note:
Look at `amount_cents`, `status`, and `customer_id`. When the agent opens this model, the relevant schema metadata is already beside the code.
*Point to the annotation, then the two `--frozen` commands.*
AnnotateRb can check whether these comments would change against the verification database. That gives us a freshness check. It doesn't check every column access the agent may write elsewhere.
Invoice is our illustrative domain here. The example is shortened, and it isn't a model from Fizzy or this website. Next, let's use a small template that we can check directly.
**Technical reference:** Real annotations have separate index and foreign-key sections. Check constraints require `show_check_constraints: true`. Prepare the verification database first. When the automatic migration hook is installed, use `ANNOTATERB_SKIP_ON_DB_TASKS=1` during preparation so the hook cannot refresh annotations before `--frozen` checks for drift. Failure means the committed metadata needs refreshing; it is not verification of arbitrary SQL.
Source: https://github.com/drwl/annotaterb#usage
---
Layer 2: Static Defense
## Herb Demo: One Nested Form
broken.html.erb — five lines, no Rails app or database.
Line 4:button_to creates a second form inside form_with.
Herb 0.11.1:html-no-nested-forms at 4:2. Exit 1.
A successful HTTP response does not check this structure.
Note:
Have a look at the Delete button on line 4. What HTML will `button_to` create?
*Pause for the audience, then point from `button_to` to the enclosing `form_with`.*
It creates a form. We already have a form, so this template nests one inside the other. A test that only checks for HTTP 200 misses that structure.
*Point to the diagnostic rule and line 4.*
Herb parses the template into an abstract syntax tree, or AST: a structured representation of the markup and embedded Ruby. With its Action View support, it recognizes these Rails form helpers without making a request. These placeholder URLs need no routes, database, or Invoice application. That gives the author a concrete next step: move the Delete form outside the original form.
Let's make that change and check again.
**Demo preparation:** Before the talk, run `cd examples/phrug-herb-demo` and `npm ci`. Keep that directory open for `npm run demo` on the next slide. This is a deliberately broken fixture for a scripted demonstration, not output from a live coding agent.
**Technical reference:** The reproduction uses Herb 0.11.1, `framework: actionview`, and only `html-no-nested-forms`. With `url:` and no `model:` or `method:`, the outer `form_with` is a POST form. It is not an edit form bound to an existing Invoice. The broken fixture returns one error and exit 1; the corrected fixture returns exit 0. These results concern this fixture and rule.
Sources: https://herb-tools.dev/linter/rules/html-no-nested-forms and https://api.rubyonrails.org/classes/ActionView/Helpers/FormHelper.html#method-i-form_with
---
Layer 2: Static Defense
## Herb Demo: Fix and Recheck
Move the delete button outside the outer form:
```bash
cd examples/phrug-herb-demo
npm run demo
```
Reproduced fixtures, one rule. A small feedback cycle I'd like to explore at project scale.
Note:
The change is small: move `button_to` after the outer `<% end %>`. We now have two sibling forms. The outer form still posts to `/invoices`; the delete form still targets `/invoices/1`.
*Point to the diff, then run `npm run demo` in `examples/phrug-herb-demo`.*
The broken template returns exit 1. The corrected template returns exit 0. The useful cycle is visible: the author gets a diagnostic, changes the code, and runs the check again.
This script runs prepared fixtures. A coding agent could consume the same feedback, but we're not running an LLM in this demonstration. We've checked one concrete structural rule and can keep its output as evidence.
*Pause on the two results, then return to the slides.* Herb can also check some contracts between a partial and its caller.
**Demo fallback:** If the terminal is unavailable, use the displayed diff and exit results to walk through the same check. Package installation belongs before the talk; neither installation nor this one-file scan measures application-wide runtime.
**Technical reference:** The demo pins `@herb-tools/linter` to 0.11.1. The underlying broken-file command is `herb-lint --only html-no-nested-forms --format json broken.html.erb`. `bundle exec herb lint` delegates to the npm linter, which also needs a pinned version.
Example `.herb.yml` for the rules discussed across these slides:
```yaml
framework: actionview
linter:
rules:
html-no-nested-forms: { enabled: true }
actionview-no-strict-locals-error: { enabled: true }
```
The demo command selects only `html-no-nested-forms`. Actual diagnostic, abbreviated:
```json
{
"offenses": [{
"filename": "broken.html.erb",
"code": "html-no-nested-forms",
"severity": "error",
"location": { "start": { "line": 4, "column": 2 } }
}],
"completed": true,
"clean": false
}
```
This excerpt preserves the real field names and location. It omits the message, end location, summary, timing, and other fields.
Source: https://herb-tools.dev/projects/linter#json-output-format
---
Herb Rule actionview-no-strict-locals-error: Checks missing or unknown locals for statically resolvable calls. Here, invoice: is required. Its Ruby type is not checked.
Note:
The partial asks for `invoice:`. The caller supplies `order:`.
*Point between those two names. Pause.*
With Rails strict locals and a statically resolvable call, Herb can report that mismatch before we exercise the page in a browser. Here `invoice:` is a required keyword argument. Fix the call with `render "invoices/row", invoice: @invoice`. We still need other checks for what that object can do.
The same idea applies beyond templates: declare a boundary, then check calls across it.
**Technical reference:** Strict locals are available in Rails 7.1+. The `locals:` declaration is the first line of the actual partial, followed by a blank line. The filename above the code is a slide label. This example declares only `invoice`, which the template uses. Herb needs the project's partial index. This example uses a literal render path and explicit arguments. Dynamic paths, splatted locals, variable hashes, and some object/collection render forms are skipped. The check does not establish that `invoice` is an Invoice or that `invoice.number` exists.
Source: https://herb-tools.dev/linter/rules/actionview-no-strict-locals-error
---
Layer 2: Architecture
## Archspec: Contracts with 'because:'
Illustrative contracts for email delivery and a public billing API:
```ruby
# Archspec.rb
component :controllers, in: "app/controllers/**/*.rb"
component :billing, in: "app/domains/billing/**/*.rb"
# 1. Reject direct synchronous email delivery calls
controllers.cannot_call :deliver_now, :deliver_now!,
because: "HTTP controllers must queue emails via deliver_later"
# 2. Expose the facade; keep other billing constants private
billing.public_api constants: "Billing::Facade",
because: "Other components must use the billing facade"
```
Run the declared checks:
```bash
bundle exec archspec check --format json
```
Note:
This first rule rejects direct calls to synchronous email delivery methods in controller code. The second gives other components a public entry point into billing.
*Point to `cannot_call`, then `Billing::Facade`, then `because:`.*
The rule determines what is allowed. `because:` tells the person or agent why we chose that rule. When a check fails, the feedback includes the architectural reason for changing the code.
The facade matters here: we want to allow that entry point while keeping the other billing constants private.
*Pause.* There is another kind of boundary that is easy to overlook: state shared between threads.
**Technical reference:** `billing.public_api` allows external references to the declared facade and rejects other billing references that Archspec can resolve. A blanket `controllers.cannot_use :billing` would forbid the facade too. The email rule checks direct calls with the configured names; it does not prove that every email is queued or follow every call through delegated services. Dynamic Ruby limits static analysis; passing means the declared checks passed on the analyzed code.
Sources: https://archspecrb.dev/rules/methods/, https://archspecrb.dev/rules/dependencies/, and https://archspecrb.dev/configuration/
---
Layer 2: Concurrency
## rubocop-thread_safety: Puma & Solid Queue
Agent wants to disable a callback temporarily during an import:
```ruby
# app/services/import_users.rb
User.skip_callback(:save, :after, :sync_profile)
user.save!
User.set_callback(:save, :after, :sync_profile)
```
The Concurrency Trap: Mutating ActiveSupport callbacks changes class state. Another thread in the same Ruby process may save a user while the callback is disabled.
```yaml
# .rubocop.yml
plugins: [ rubocop-thread_safety ]
ThreadSafety/ActiveSupportCallbacks: { Enabled: true }
```
Note:
Imagine another request saving a user between these two callback calls.
*Point to `skip_callback`, then the gap before `set_callback`. Pause.*
The callback configuration belongs to the class. Another thread in that process may save while the callback is disabled, even though a sequential test looks fine. This can happen among Puma request threads or within a Solid Queue worker process; separate processes have their own copies of the class state.
This cop flags that mutation directly. When one particular save should skip optional work, a per-instance condition is a better direction to investigate.
We've looked at checks that inspect the code. Now let's look at behavior we need to execute.
**Technical reference:** Use rubocop-thread_safety 0.8 or newer. `ThreadSafety/ActiveSupportCallbacks` is pending, so explicitly set `Enabled: true`. The cop catches this pattern; it does not establish general thread safety. **Rehearsal checkpoint:** finish schema/static checks and the Herb demo around 21:00.
Source: https://docs.rubocop.org/rubocop-thread_safety/cops_threadsafety.html#threadsafetyactivesupportcallbacks
---
Layer 3: Runtime Defense
## Prosopite: Runtime SQL Fingerprinting
Illustrative query loop outside association loading:
```ruby
orders.each do |order|
Customer.find(order.customer_id).name
end
```
strict_loading guards lazy association loading, not direct Customer.find calls.
Enable exceptions, then scan an integration test:
```ruby
# config/environments/test.rb
config.after_initialize { Prosopite.raise = true }
# Inside an ActionDispatch::IntegrationTest subclass
test "orders list has no detected N+1" do
Prosopite.scan { get orders_path }
end
```
Note:
Look closely at `Customer.find` inside the loop. This is a separate query, rather than lazy loading through `order.customer`.
This is the N+1 pattern: one query loads the orders, then another query runs for each order. A SQL fingerprint represents a query's shape without the individual values. Prosopite combines repeated query shapes with their call stacks to look for this pattern.
*Point to the query, then `Prosopite.raise = true`, then the scan.*
Prosopite observes the SQL from the exercised request. Setting `raise` makes a detection fail the test, so the agent receives feedback it has to act on.
The test data matters. We need multiple orders with distinct customers to exercise this case. A passing scan gives us evidence about that execution, not every endpoint in the application.
Next, let's move from repeated queries to a database invariant.
**Technical reference:** The illustrative example needs an orders endpoint, fixtures with at least two orders belonging to distinct customers, and any required authentication. Put the initializer in `config/environments/test.rb`; put the test in a separate integration test file requiring `test_helper` and subclassing `ActionDispatch::IntegrationTest`. Prosopite uses `sql.active_record` events, query fingerprints, and call stacks. Detection does not raise by default, and the check covers executed paths and matching queries.
Source: https://github.com/charkost/prosopite#test-environment-usage
---
Layer 3: Database Integrity
## DatabaseConsistency & Strong Migrations
Unique number per account. Prerequisite: `account_id` and `number` are already **NOT NULL** in the DB.
```ruby
class Invoice < ApplicationRecord
validates :number, presence: true, uniqueness: { scope: :account_id }
end
```
The Race: Concurrent writes can bypass model uniqueness validation. Add a matching unique index.
**DatabaseConsistency** reports missing indexes; **Strong Migrations** checks supported DDL risks.
```ruby
# Existing PostgreSQL table; inside a migration class
disable_ddl_transaction!
def change
add_index :invoices, [:account_id, :number], unique: true, algorithm: :concurrently
end
```
Note:
The product rule is that an invoice number is unique within an account. This example assumes the database already requires both the account and the number.
*Point to `scope: :account_id`, then the matching columns in the index.*
The Rails validation is useful, but concurrent writes still need the database to enforce that rule. DatabaseConsistency checks whether the matching unique index exists. The index enforces uniqueness of the pair; the existing NOT NULL constraints make both values mandatory.
Creating that index is a separate concern. For this existing PostgreSQL table, we use a concurrent index and disable the migration transaction. Strong Migrations can check documented unsafe DDL patterns.
*Pause.* We still need to decide what happens if the migration is rolled back.
**Technical reference:** `MissingUniqueIndexChecker` compares validations with indexes; it does not make a migration safe. A normal PostgreSQL unique index treats NULL values as distinct, so nullable columns can admit duplicates that the intended invariant would forbid. Rails `presence: true` does not create a database NOT NULL constraint. Both columns must already have that constraint for the example's stated invariant. If they do not, handling existing NULLs and adding the constraints need a separate migration plan; this snippet only adds the index. Nullable designs need an explicit decision about NULL semantics.
The shown body belongs inside an `ActiveRecord::Migration` subclass. `algorithm: :concurrently` avoids the ordinary index build's write-blocking behavior and requires `disable_ddl_transaction!`. Existing duplicates must be addressed first. The recipe does not transfer unchanged to SQLite or MySQL. Strong Migrations does not detect dangerous backfills, and neither tool establishes zero downtime for a complete deployment.
Sources: https://github.com/djezzzl/database_consistency, https://github.com/ankane/strong_migrations#adding-an-index-non-concurrently, and https://www.postgresql.org/docs/current/indexes-unique.html
---
Layer 3: Runtime Defense
## Rollback Is a Smoke Test
| Tool | What Question It Answers | When It Runs |
|------|--------------------------|--------------|
| **AnnotateRb** | What is the schema of this model? | Context in model, `--frozen` in CI |
| **DatabaseConsistency** | Do supported model/DB checks pass? | Boots the app: `database_consistency` |
| **Strong Migrations** | Does this DDL trigger a known unsafe pattern? | Up by default; down is opt-in |
```ruby
# config/initializers/strong_migrations.rb
StrongMigrations.check_down = true
```
```bash
# DATABASE_URL must point to an isolated, disposable DB.
RAILS_ENV=test bin/rails db:migrate &&
RAILS_ENV=test bin/rails db:rollback STEP=1 &&
RAILS_ENV=test bin/rails db:migrate
```
What passing means: The last migration ran down and up without an error. It does not prove data preservation or production safety.
Note:
This command answers a small but useful question: can the last migration run down and then up again?
*Point to `check_down = true` before the shell commands.*
Strong Migrations checks upward migrations by default. This initializer opts into its configured checks during rollback too. It checks supported DDL risks; preserving the data still needs assertions.
*Point to `STEP=1`. Pause.*
If a change includes several migrations, we need to test the intended versions. And a successful rollback can still lose valuable data. Representative data and assertions about its preservation are what make that property testable.
The connection target comes first: run this only on an explicitly provisioned disposable database.
Now let's follow a request all the way into the browser.
**Technical reference:** Put `StrongMigrations.check_down = true` in the initializer of the app being checked, with the gem loaded in that environment. The setting enables the configured checks during down migrations; it does not enable every optional rule or verify data preservation. The previous slide's reversible `add_index` retains `algorithm: :concurrently` for its inverse in Rails 8.1.4, so it does not need a separate `down` method.
Prepare the baseline schema and representative data in an isolated verification database, and verify the connection target. `RAILS_ENV=test` is not isolation; `DATABASE_URL` can override the destination. `STEP=1` covers the latest migration only, even if this change added no migration. A full verifier must select the intended migration versions. An empty database can hide data loss, while down/up may succeed after dropping data. Add assertions for the relevant domain invariants. The example carries no measured migration failure rate.
Sources: https://guides.rubyonrails.org/active_record_migrations.html#rolling-back, https://github.com/ankane/strong_migrations#down-migrations--rollbacks, https://github.com/ankane/strong_migrations#backfilling-data, and https://github.com/rails/rails/blob/v8.1.4/activerecord/lib/active_record/migration/command_recorder.rb
---
Layer 3: Hotwire & UI
## Hotwire: HTTP Green, Frame Error
Illustrative inline-editing response with a missing matching frame:
```erb
<%# app/views/invoices/show.html.erb %>
<%= turbo_frame_tag "invoice_details" do %>
<%= link_to "Edit", edit_invoice_path(@invoice) %>
<% end %>
<%# app/views/invoices/edit.html.erb %>
<%# BUG: Agent forgot the matching turbo_frame_tag or misspelled ID! %>
<%= form_with model: @invoice do |f| %>
<%= f.text_field :title %>
<%= f.submit "Save" %>
<% end %>
```
The Trap: An HTTP 200 OK does not establish that the response contains turbo-frame#invoice_details. By default, a missing frame produces a message in the frame and a JavaScript exception.
Note:
The show page expects a frame named `invoice_details`. Now look at the edit response: where is the matching frame?
*Point to the opening frame on the show page, then the bare form in the response. Pause.*
The endpoint can return HTTP 200 while Turbo rejects this response. By default, the missing frame produces a message in the frame and a JavaScript exception.
A request test can check for the expected frame. A browser test can click Edit, fill in the form, save it, and check the updated content. That gives us evidence for the interaction we actually care about.
We've now seen why the checks need different scopes. Next, we need to protect the checks themselves.
**Technical reference:** Use a JavaScript-enabled system test and check the form within `turbo-frame#invoice_details`. The result covers the exercised scenario. Herb's structural analysis alone does not verify the whole Hotwire interaction. **Rehearsal checkpoint:** finish runtime examples around 27:30.
Source: https://turbo.hotwired.dev/handbook/frames#breaking-out-from-a-frame
---
Layer 4: Verification
## Protect the Acceptance Checks
Illustrative failure: weakening an assertion can turn a test green.
```ruby
# Illustrative weakened test:
# BEFORE: assert_equal 42, calculated_total
# AFTER: assert true
# Or in RSpec: expect(calculated_total) # No matcher attached!
```
Proposed controls for a trusted verifier:
1. **Assertion linting:** explicitly enable `Minitest/UselessAssertion`; use `RSpec/VoidExpect` for RSpec.
2. **Protected checks:** compare tests, rules, and verifier code against a reviewed baseline; review intentional changes.
3. **Patch coverage:** example threshold after collecting fresh coverage:
`simplecov patch --base origin/main --minimum 100`
Note:
Both assertions can produce a green test. Only the first one checks the calculated total.
*Point to `assert_equal 42`, then `assert true`. Pause.*
A few assertion linters can catch simple cases like this. We also need a trusted place to run acceptance checks and a reviewed path for intentional changes to those checks.
Coverage can help us see which lines ran. It doesn't tell us whether the assertions expressed the right requirement. The 100% shown here is a chosen patch-coverage threshold, rather than a measured feature success rate.
Let's put these checks into the agent's workflow.
**Technical reference:** These are proposed controls, not an implemented security boundary in this repository. Run acceptance checks from trusted CI configuration or a checkout the authoring agent cannot silently replace. Protect tests, rules, verifier code, and coverage configuration. `Minitest/UselessAssertion` is pending and needs explicit enabling; these cops detect specific weak assertions, not arbitrary manipulation. The SimpleCov example needs a version with the patch CLI, a fresh coverage report covering the intended files, and the intended base revision. Line execution does not prove assertion quality or feature correctness.
Sources: https://docs.rubocop.org/rubocop-minitest/cops_minitest.html and https://github.com/simplecov-ruby/simplecov/blob/main/docs/CLI.md
---
Layer 4: Harness Architecture
## A Proposed Agent Workflow
1. Human + agentAgree on a ticket and acceptance checks.
2. Author agentImplement the change in an isolated worktree.
3. Project checksReturn diagnostics. Repair within time and attempt limits.
4. Reviewer agentInspect the diff and evidence with fresh context.
5. Human reviewAccept the outcome, permissions and data behavior.
The demo's loop, at project scale: A failed check returns to the author. A persistent failure or an unclear requirement returns to a human.
Note:
One workflow I'd like to try starts with an agreed ticket and acceptance checks. The agent writes a change, runs the checks, and uses failures to guide another attempt. I'd be interested in how your teams structure that loop.
*Trace the loop, then stop at Human Review.*
We put a limit on attempts or elapsed time. A persistent failure or an unclear requirement comes back to a person. A fresh reviewer agent can add scrutiny, but it can also miss the same thing as the author.
The human acceptance step asks whether the requirements and risks are covered by the evidence. That evidence must identify the exact source state we tested, including any uncommitted changes.
Before we run several agents like this, we need to separate their runtime state.
**Technical reference:** This workflow is proposed rather than a measured autonomous delivery result. Rerun affected checks after changes. Acceptance checks and the human decision remain necessary; weakening a gate is not a valid way to finish the repair loop.
---
Harness Infrastructure
## Worktree & State Isolation
Parallel worktrees also need separate runtime resources:
| Resource | Isolation Strategy | Gotcha to Avoid |
|----------|-------------------|-----------------|
| **Source Code** | Git worktree | Code isolation does not isolate services |
| **Database** | Separate SQLite path or server DB name | Shared connection URL |
| **Solid Queue** | Separate queue database when enabled | Workers polling the same queue |
| **Action Cable** | Separate Redis prefix or Solid Cable database | Cross-talk between test runs |
| **File Storage** | Separate local directory or bucket/prefix | Shared blobs |
If two workers poll the same queue database and queues, either can execute the other's jobs.
Note:
A worktree gives an agent its own source checkout. We also need to look at what the application connects to.
*Point to the database row, then Solid Queue.*
If both workers poll the same queue database and queues, either can execute the other's jobs. The same question applies to storage, Action Cable, ports, and browser sessions.
This depends on configuration. Rails-root-relative SQLite files may already be separate; a shared connection URL can still send both worktrees to one service. Check the actual destinations.
Once those resources are isolated, one project command can coordinate the verification.
**Technical reference:** This is a configuration-dependent scenario. Solid Queue must be enabled in the relevant environment; standard Active Job tests often use the test adapter. A custom `AGENT_STATE_DIR` variable needs explicit wiring into database and storage configuration because Rails does not interpret it automatically. Isolate ports and browser sessions for concurrent system tests too.
Sources: [Solid Queue setup outside production](https://github.com/rails/solid_queue#usage-in-development-and-other-non-production-environments), [Rails Active Job](https://guides.rubyonrails.org/active_job_basics.html#default-backend-solid-queue).
---
Harness Design
## A Proposed Interface: `bin/verify`
One project command can coordinate checks and collect their results:
```bash
# Proposed commands; this repository does not implement them yet
bin/verify --fast # annotation freshness + template/architecture checks
bin/verify --full # fast checks + DB/migration + SQL + system tests
```
Required behavior: Fail on failed or missing checks, preserve diagnostics, and identify the exact code tested. Measure runtime before setting budgets.
Note:
This is the interface I'd like the agent to have: one command for short feedback, and one for the full acceptance run.
*Point to `--fast`, then `--full`.*
The runner should collect the diagnostics, return failure if a required check fails or never runs, and identify the exact code tested. That includes missing tools, crashes, and timeouts.
A commit SHA alone isn't enough when the agent has uncommitted changes. We can verify a clean checkout of that commit, or capture the working-tree changes and untracked source files too. If the code changes, the previous result no longer describes the new version.
There is no `bin/verify` implementation in this repository yet. This slide specifies what we would build around the individual checks. We would then measure its runtime and check that local and CI runs use the same versions and configuration.
So, what can a green result actually tell us?
**Technical reference:** Prepare an isolated database and prevent annotation preparation from auto-fixing drift. Run all required tools and propagate nonzero exits. Strong Migrations runs through migrations; Prosopite wraps selected scenarios. Run the normal suite and browser/system tests explicitly. JSON aggregation, command logs, identity of the complete source state, repair limits, and verified local/CI parity are implementation requirements. Keep that source state stable during verification and rerun affected checks after changes. No full-pipeline runtime has been measured.
---
Mechanical Reliability
## What Each Check Establishes
For each result, I'm asking what it establishes and what remains outside the check:
| Check | Evidence | Limit |
|-------|----------|-------|
| **AnnotateRb** | Annotations match the checked DB | Does not validate every column access |
| **DatabaseConsistency** | Configured model/schema checks pass | Does not prove business invariants |
| **Strong Migrations + rollback** | Checked DDL and rollback execute | Data preservation needs assertions |
| **Herb + system tests** | Static rules and tested UI flows pass | Untested interactions remain |
| **Archspec** | Analyzed references obey declared rules | Dynamic dependencies need care |
| **Prosopite** | No N+1 detected in scanned execution | Other paths and data remain untested |
Benchmark impact: The effect of this proposed pipeline on Fizzy's mean task score has not been measured.
Note:
Read this table across a row: the check, the evidence it gives us, and its limit.
*Point to the Prosopite row, then the Herb and system-tests row.*
“No N+1 detected in this execution” is a useful result. “This browser flow passed” is another. Those statements tell the reviewer exactly what was exercised.
We have published benchmark results, a reproduced Herb example, and documented checks that we could integrate. We don't yet have a before-and-after Fizzy result for this pipeline. Measuring that would be a separate experiment with the same tasks and model settings.
*Pause.* That leaves some useful questions for the community.
**Technical reference:** No percentage of entire failure classes eliminated has been measured. Deterministic rules still have limited scope. A comparison of the implemented pipeline would need a fixed task set and model settings, preserved logs, and outcomes with and without the added checks. **Rehearsal checkpoint:** finish the verification workflow around 33:00.
---
Community Discussion
## Open Questions for the Rails Community
Questions I'd like to compare notes on:
1. **Experience:** Which checks have caught useful problems, and which mostly added friction?
2. **Evidence:** How do you tell whether an agent workflow improves delivery? What would change your mind?
3. **Shared tools:** What belongs in each project, and what could Rails provide by default?
Let's exchange examples: Failed attempts and counterexamples can help us decide what to test next.
Note:
These are questions I don't have settled answers to. Which checks are worth maintaining? How do we measure their effect? What could Rails standardize?
*Point briefly to each question; save audience discussion for the final slide.*
An official `rails-harness` is one possible direction to discuss. Your team may need a different combination of checks or a simpler workflow. I'd like to hear those examples too.
For my capstone, I'd like to compare the effort of maintaining these checks with the rework they prevent. Which trade-offs have you seen in your own projects?
Let me leave you with the ideas I'm taking into my next experiments.
**Reference:** These are questions for future work and community discussion. This talk does not report a measured reduction in rework.
---
Summary
# Ideas I'm Taking Forward
1. Clear acceptance criteria: I want to make “done” explicit and try a workflow proportional to the change.
2. Feedback the agent can use: I want to connect project requirements to executable checks.
3. Measured outcomes: I want to learn which checks improve delivery, what they miss, and what they cost.
The open question: Which combination helps on real tasks? I'd like to find out together.
Note:
These are the three ideas I'll test next. First, make the acceptance criteria explicit and keep the process proportional to the change.
*Reveal the next takeaway.*
Second, explore which instructions can become useful executable feedback: schema checks, template rules, architecture rules, and scenarios that exercise the application.
*Reveal the final takeaway, then pause.*
Third, measure outcomes and keep the limits visible. The benchmark gives us observations. Our project checks establish specific properties. The benefit of combining them needs measurement. Human review still matters for requirements, money, permissions, and data integrity.
I'd like to compare these choices with your experience, especially examples that challenge the approach.
**Reference:** The talk presents a proposed architecture and illustrative snippets, not a completed pipeline or measured improvement in feature delivery. **Rehearsal checkpoint:** finish the takeaways around 35:00 and open Q&A.
---
Questions & Discussion
# Let's Compare Notes
Salamat, PhRUG!
What has worked for you? What would you try differently?
Stay in touch on LinkedIn
I write about Ruby and software development, and share what I'm learning.
Note:
Salamat, PhRUG! Thank you to the organizers and everyone here.
*Point to the QR code and leave this slide visible during the discussion.*
If you'd like to stay in touch, please follow me on LinkedIn. I write about Ruby and software development, and I'll share what I'm learning as this work develops. The QR code opens my profile. I'd also love to hear about your experiments, including things that didn't work.
The Ruby Overdrive link has the slides, sources and code examples. To start the discussion: which check has caught a real problem in an agent-generated change on your project?
*Pause and give the audience time to respond, then invite questions about the talk. Keep the broader community questions as prompts if the room needs another starting point.*
**Reference:** LinkedIn profile and the original QR image were supplied by the speaker. Keep the QR code's white border and square proportions intact.
**Rehearsal plan:** use approximately 35:00–40:00 for questions within the 40-minute slot. This is a proposed schedule, not a measured rehearsal result. The race widget uses the full 40-minute duration and does not reserve Q&A automatically. If time is running short, move to questions after the takeaways rather than opening a separate discussion on the community-questions slide.
