Building Shelf

What it is

Shelf lets a shop owner manage their Shopify catalogue by typing what they want. Rewrite the Rega Planar 3. Put the Entreq Olympus range to draft on Friday. What's failing an SEO check? Shelf drafts it, shows a preview, and does nothing until the merchant approves. Every write is reversible.

I run ecommerce and marketing for an independent hi-fi shop in Norwich, with a catalogue of about 1,150 products. The work Shelf does is work I was doing by hand. It has been developed against that live store throughout — real products, real customers, real damage when it got things wrong.

I am not a trained software engineer. What I have is ten years of running the catalogue this application writes to, and AI coding tools let me turn that into working software. For an application of this kind that is the relevant fact rather than an embarrassing one: the interesting question is not whether a model can produce code, it is whether the person directing it can tell when the output is quietly wrong. Most of what follows is about that.

By the numbers

Server code

16,858 lines JS

Test cases

662 0 failing

Catalogue

1,153 products

SEO rules

23

Models

2 Sonnet + Haiku

The thesis

Every serious bug in five months was an ordinary systems bug wearing an AI costume.

Not one of them was fixed by changing a prompt. An index misaligned in a batch loop. A tokeniser blind to a forward slash. A fallback that returned the whole candidate list when it recognised nothing. A language model handed the last word on whether it had actually done the thing it said it had done.

The prompt file already carried five escalating instructions never to invent a specification, and it invented them anyway. That is the lesson I would bring to any team building on top of a model: the prompt is where you express intent, and it is almost never where you enforce it. Enforcement is code — deterministic, testable, and cheap.

Four incidents make the case. All four are real, all four are in the git history, and two of them damaged live customer-facing data before I caught them.

Four incidents

Making it cheap

A clean product rewrite originally made six model calls and cost roughly £0.20–0.30. Two brand ranges came to about $20. The obvious lever was the specification handling, which was also the accuracy problem — the model was writing figures, then three further calls were spent asking it to check its own figures.

Both problems had one answer: stop the model writing specifications at all. Code lifts the manufacturer's specification block verbatim and pastes it in. The model writes prose only, and prose is forbidden any figures. A deterministic scan extracts every number-and-unit from the draft and strips any that isn't character-for-character in the source. No model call, no cost, and — unlike an instruction — it cannot be ignored. Four of the six calls disappeared.

I built the cost accounting first, so the saving could be measured rather than assumed: an async-local bucket per HTTP request, the Anthropic client wrapped once so no call site escapes it, cache reads and writes priced separately. Web search bills per search rather than per token, so that rate is read from an environment variable and never guessed — searches are still counted, so the count stays trustworthy even when the rate isn't set.

Separately, the system prompt was split into three cached blocks, with the large writing-rules block sent only when the request is actually a writing request. Non-writing calls — price checks, stock questions, ordinary conversation — dropped from about 5,500 tokens to about 1,750.

What I test, and why

662 cases across 30 files, green as of today. Three of the rules behind them were bought with damage.

• 01

  Skipped is never passed. The SEO ruleset reports a check it couldn't run as SKIPPED and excludes it from the score denominator. The catalogue sweep doesn't fetch images, and silently passing the image rules would have rebuilt exactly the false all-clear the ruleset was written to remove.

• 02

  A test file CI doesn't run is not a test. The regressions that let SHF-BH35 through were sitting in a file the npm test script never invoked. The suite was green and the suite meant nothing. Worse, an existing test named "no match falls back to full fuzzy list" never exercised the no-match branch at all — its fixture matched both products.

• 03

  When a matcher gets more permissive, the negative test is the one that matters. The first cut of the BH35 fix introduced two new bugs by widening the match: "Rega Planar 3" began sweeping "Rega Planar 3/24V". Every test I wrote first time asked whether it found the right product. None asked whether it now also found the wrong one.

On top of the unit tests there is a replay harness built from real failures — each named for the incident it came from, run against the actual handler rather than a mock. A separate pipeline audit runs a corpus of real products end to end before a release, so the long tail surfaces in one batch instead of one live failure a day. Its first free run found five manufacturer sites that block automated fetches — not bugs, but the difference between Shelf saying "I couldn't find specs" and "that site blocks us, please paste them".

What isn't done

The version of this document that claimed otherwise would be less useful to anyone reading it.

Where this leaves me

I set out to solve a specific problem in my own shop and ended up with a reasonable education in why systems built on language models fail in production. The failures were not exotic. They were fallbacks, off-by-ones, unread return values and a component trusted to report on itself — the same faults as any other software, made harder to spot because the output is fluent prose that reads like success.

What I would want to be judged on is not the code volume. It is that every incident above was found, traced to an actual root cause rather than a symptom, fixed, tested against the case that would have caught it, and written down.