Testing

Quality by design: acceptance criteria and invariants exist before the code. The API is where everything gets proven.

The API is the test boundary

All business logic sits behind the API. That single rule makes the whole system testable without a browser, and gives agents a fast feedback loop.

Fig. 8 Because the frontend holds no logic, end to end means API to database. A handful of browser tests cover the UI itself.

Test levels

Level	What	How
Unit	Parser, compiler, validator, expressions	Vitest
Function contracts	Every function against its schema and idempotency	Contract tests with fake SMTP and PDF
Definitions	The application definition loads and all references resolve	`core validate` in CI
Processes	Declarative: given, when, expected steps, records and outputs	Runner in test mode, step by step
API integration	Generic endpoints against real PostgreSQL	Testcontainers
Permissions	Every role and scope: allowed and denied, including partners	API tests with different accounts
Contract	No unintended breaking changes	OpenAPI diff as a CI gate
Invariants and end to end	Reality map level 5 scenarios	Black box against the API
Frontend	Display and input only	A few Playwright tests

Invariants test the whole

Modules can pass every test while the system as a whole drifts. Invariants are business rules that must hold at any time, checked after every scenario.

Stock movements and stock value match the documents
Open items equal invoices minus payments minus credits
No negative available stock, ever
A partner never sees another partner's data

// Must hold after every scenario, for every customer.
export const openItemsBalance = invariant("open items equal invoices minus payments minus credits",
  async (api) => {
    for (const c of await api.query("customer")) {
      const invoiced = await api.sum("invoice", "total", { customer: c.id })
      const paid     = await api.sum("payment", "amount", { customer: c.id })
      const credited = await api.sum("creditNote", "total", { customer: c.id })
      const open     = await api.sum("openItem", "amount", { customer: c.id })
      expect(open).toEqualMoney(invoiced.minus(paid).minus(credited))
    }
  })

What gets measured

The same metrics for every framework and model. A baseline run without any framework shows what the frameworks actually add.

Acceptance score

weighted share of passing acceptance tests

Reality map

status per check, five levels

Prompts and interventions

every time I step in, with a reason

Self correction rate

errors agents find and fix on their own

Time

wall clock and my active time

Cost

tokens and subscriptions

Code quality

lint, coverage, static analysis, security scan

Changeability

prompts needed for the same change request

Run	Workflow	Builder	Reviewer	Status
1	Flow-Next	Claude	other model family	next
0	No framework, baseline	same model	none	planned
2	GitHub Spec Kit	tbd	tbd	planned
3	Superpowers	Claude	tbd	planned
4	BMAD Method	tbd	tbd	planned
CR	OpenSpec, for the change request test	tbd	tbd	planned

Fair play: same specs, same tests, same time box, tool versions and model IDs recorded, at least one run repeated to show variance.