ProductSpec Trial
I gave two AI agents the same product idea. One spec had a Risks section. Here's what changed.
There’s a new open standard called ProductSpec by the awesome Gokul Rajaram (released this month). It’s a Markdown format for writing product intent (the problem, the hypothesis, the scope, the acceptance criteria, the success metrics) in a structure that’s readable by humans and buildable by AI agents. The pitch is that a Product Spec should sit upstream of engineering specs, tickets, and code: the durable record of why something is being built, written once, versioned as it changes, and handed off to whoever (or whatever) builds it next.
That “buildable by AI agents” claim is the part worth testing, not just reading about. So I picked a real idea, wrote it two ways, and had two AI agents (from the same foundational model) build from each version with zero shared context between them.
The idea
A portal that shows the nearest blood banks and their likely blood supply for people urgently in need. Before writing anything, I checked whether this was buildable cheaply. Location data is easy: OpenStreetMap tags blood banks directly, and it’s well populated in India specifically, sourced from government open data. Live blood supply is the hard part. India has eRaktKosh, a public government search tool showing blood stock by state, district, and blood group, self-reported by blood banks. But it’s a web page, not a documented open API. Outside India, nothing like it exists (as far as I know); blood donation orgs publish donation locations, never inventory. So the idea has a real, resolvable location layer and a genuinely unresolved supply layer. That gap turned out to be the most useful part of the whole experiment.
Two versions of the same spec
I wrote the same Product Spec twice. Same problem, same hypothesis, same scope, same acceptance criteria, same success metrics. The only difference:
Full version includes ProductSpec’s optional Risks and Open Questions sections, naming the thing I already knew was shaky: eRaktKosh’s API access terms are unconfirmed, and if it turns out to require a partnership request rather than a self-serve key, the supply feature ships late or not at all.
Minimal version has only the five sections ProductSpec marks as mandatory: Problem, Hypothesis, Scope, Acceptance Criteria, Success Metrics. No Risks section. No Open Questions. The eRaktKosh uncertainty is nowhere in the document.
Both files validate cleanly against ProductSpec’s own CLI. Both are legitimate Product Specs by the standard’s rules. The only difference is whether the one thing I was worried about was written down.
(Spec files: bloodbank-locator-full.product-spec.md,
bloodbank-locator-minimal.product-spec.md)
Round one: kickoff plans
Before building anything, I gave each spec to a fresh AI agent with no other context and asked for an engineering kickoff response: what to build first, what assumptions it was making, what it would flag as unresolved.
The agent working from the full spec produced a sensible plan that worked around the risk I’d already named. It sequenced the build to avoid depending on eRaktKosh early. Useful, but it didn’t go further than what I’d already told it to worry about.
The agent working from the minimal spec, with no Risks section at all, still caught the eRaktKosh problem on its own. It read one line in the acceptance criteria describing graceful degradation if the data source failed, and inferred from that alone that the source was unreliable enough to plan for. Then it went further and found something I hadn’t written down anywhere: OpenStreetMap and eRaktKosh have no shared identifier, so there’s no defined way to match the same physical blood bank across the two data sources. It called this “likely the riskiest part of the build.”
That was the first sign that an explicit Risks section might not be doing the job I assumed it was doing. A well-written Acceptance Criteria section leaks risk to a competent reader whether or not you spell it out separately. And naming your own risks might anchor the reader on your blind spots instead of prompting them to look for new ones.
I went ahead with the build nonetheless and found something interesting.
Round two: actually building it
I had each agent build a real, working v1: a single self-contained HTML file, no build step, using the free OpenStreetMap Overpass API for live blood bank search, with instructions not to call or scrape eRaktKosh (its access is unresolved) and to use its own judgment on how to handle the supply-status feature given that constraint. Each agent worked in an isolated folder with no knowledge the other spec or build existed.
Both agents built working search tools. Both got geolocation, distance sorting, radius filtering, and one-tap calling right. And both made the same fundamental judgment call: neither one faked a live eRaktKosh feed. That constraint held regardless of how much the spec spelled out.
But the two builds disagreed on what to show by default.
The full-spec build (Risks section present) defaults every result to “Unknown, call to confirm.” Sample supply data exists, but only behind an explicit, off-by-default toggle labeled “Preview supply-status UI with sample data,” with a loud “SAMPLE, NOT REAL” badge. Nothing resembling live data appears unless you go looking for it.
The minimal-spec build (no Risks section) defaults to showing a “simulated” status on roughly a third of results automatically, amber-colored, clearly labeled “(simulated),” deterministically generated so it’s stable across searches rather than flickering. Still honest, still labeled. But on by default instead of opt-in.
Same underlying constraint. Same honesty. Different default posture, and the difference traces back to one optional section in the source document.
What I think ProductSpec should reconsider
ProductSpec’s stated design principle is to structure the parts machines must execute and leave the parts humans must reason about readable. Under that logic, Risks and Open Questions are optional because they’re reasoning, not execution: they don’t map to a fenced, machine-parseable block the way Scope and Success Metrics do.
But that logic breaks down the moment the reader isn’t a human anymore. A human PM reading a spec without a Risks section will ask “wait, what about eRaktKosh?” before a single line of code gets written. An AI agent building autonomously doesn’t ask. It makes a reasonable, defensible, unreviewed judgment call and ships it, as both of these prototypes just demonstrated. The minimal spec’s build didn’t fail or refuse to build the ambiguous part. It quietly decided, on its own, to show unverified data to users by default in a tool people might use during a medical emergency. That decision was sound, even careful, but it was made by an agent, alone, precisely because there was nowhere in the document for me to have made it first.
That’s the actual case for making Risks and Open Questions mandatory in any spec meant for agent execution, not just nice-to-have. They’re not documentation of your reasoning for a human’s benefit. They’re the mechanism by which a human’s judgment call survives into a build a human isn’t watching in real time. Leaving them optional treats agent-executed specs the same as human-executed ones, and the whole premise of ProductSpec is that they’re not the same thing anymore.
Both prototypes, both spec versions, and the full build READMEs are public: github.com/shri816/productspec-bloodbank-trial. Open either index.html directly in a browser to try it (give the Overpass API a few minutes to cool down if search fails).