One pass of the cycle
A pass begins with a scheduled GET and ends with a JSON receipt. Between those two points the application resolves a bank key, upserts accounts, walks every transaction page, and writes what it has not seen before.
← Scroll to see the full diagram →
2024-01-01 on every run,
so “incremental” describes the write behaviour rather than the read: each
pass re-reads the entire history and relies on the upsert to make that cheap. And
runFullSync() is a stub whose whole body calls
runIncrementalSync(), with a comment acknowledging it.
Two files, one specifier
The repository contains both src/server/sync.ts and
src/server/sync/index.ts. Both export runIncrementalSync,
runFullSync and getSyncStatus. Only one of them ever runs.
Under the module resolution this project uses — tsconfig.json sets
"moduleResolution": "bundler" — the bare specifier
@/server/sync resolves to the file sync.ts in preference
to the directory's index.ts. Every production import uses the bare specifier.
Every test that exercises the categorisation trigger uses the deep path
@/server/sync/index.
← Scroll to see the full diagram →
src/server/actions/settings.ts reads
result.transactionsSynced and result.newTransactions, which
exist only on sync.ts's result type; and /api/sync serves the
getSyncStatus() shape that sync.ts returns, not the
{ lastSyncTime, totalTransactions, uncategorizedCount, accountBalances }
shape the directory version returns.
Nothing is broken in a way a user would see. Transactions still sync, and they are still
classified — by the independent */10 categorisation cron, which was
added in TASKS.md Phase 8 as a “backup categorization pass”. The
practical effect is that the backup became the only pass, and the worst-case delay
between a transaction arriving and being categorised is ten minutes rather than the few
seconds the sync-triggered path was designed to give. The more interesting cost is to
assurance: ten tests certify a behaviour that never executes.
The Mercury client
src/lib/mercury/ holds a typed client: a class, a singleton accessor, a reset
function for tests, and a structured error. It is a careful piece of work — and the
live sync path does not use it. src/server/sync.ts calls
fetch directly against the same base URL. The client is reached only through
the shadowed sync/accounts.ts and sync/transactions.ts.
- new MercuryClient(key, opts?)
- Throws
'Mercury API key is required'on an empty or whitespace-only key. Options acceptmaxRetries,retryBaseDelayandtimeout. - getBaseUrl()
- Returns
https://api.mercury.com/api/v1, exposed so a test can assert it without reaching into a private field. - getAccounts() / getAccount(id)
/accountsand/account/{id}. The single-account call rejects with'Account ID is required'for a blank id, before any network call.- getTransactions(id, opts?)
/account/{id}/transactions. Query parameters arestart,end,limit,offsetandstatus.- getAllTransactionsForAccount(id, since?)
- Pages at
pageSize = 500, incrementingoffset, until a short page arrives or the accumulated count reaches the reportedtotal. - getMercuryClient()
- Lazy singleton over
MERCURY_API_KEY; throws'MERCURY_API_KEY environment variable is not set'when unset.
PRD.md specifies “Cursor-based pagination for all transactions”,
and src/lib/mercury/types.ts still carries a cursor field on the
params interface and a MercuryPaginationCursor type to match. Neither is
referenced anywhere. Both the client and the live sync walk a numeric offset. The cursor
types are dead surface left behind by the specification.
Transport settings
| Setting | Value | Behaviour |
|---|---|---|
maxRetries | 3 | Four attempts in total for a retryable failure |
retryBaseDelay | 1,000 ms | Doubled per attempt: base × 2attempt |
| Jitter | 0–30% | Math.random() * 0.3 * baseDelay added to each wait |
Retry-After | honoured | When present it overrides the computed backoff |
timeout | 30,000 ms | Enforced with an AbortController; an abort becomes MercuryError('Request timeout', 408, 'TIMEOUT') |
| Concurrency | 1 | Accounts are walked serially |
The synthesised 408 is worth noting: because isRetryable() covers only 429 and 5xx, a client-side timeout is not retried, while a server-side 503 is.
Error taxonomy
| Status | isAuthError | isRateLimited | isRetryable |
|---|---|---|---|
| 400 | false | false | false |
| 401 | true | false | false |
| 403 | true | false | false |
| 404 | false | false | false |
| 429 | false | true | true |
| 500 | false | false | true |
| 503 | false | false | true |
Every cell is a separate assertion in __tests__/lib/mercury/client.test.ts. An authentication failure is explicitly not retryable, so a revoked key fails fast rather than consuming the retry budget.
Running the client suite
The Mercury tests are split. Twenty-seven unit tests always run; six integration tests make
real API calls and are gated behind three conditions — a key must be present, native
fetch must exist, and SKIP_MERCURY_INTEGRATION must not be set.
When they are skipped the suite prints the reason rather than staying silent.
$ npm test -- __tests__/lib/mercury/client.test.ts > gogenticcfo@0.1.0 test > jest __tests__/lib/mercury/client.test.ts console.log Integration tests: SKIPPED (MERCURY_API_KEY not set) at Object.<anonymous> (__tests__/lib/mercury/client.test.ts:352:7) PASS __tests__/lib/mercury/client.test.ts MercuryClient constructor ✓ should create a client with a valid API key ✓ should throw an error when API key is empty ✓ should throw an error when API key is whitespace ✓ should accept custom configuration getBaseUrl ✓ should return the correct base URL MercuryError isRetryable ✓ should return true for 429 status ✓ should return false for 401 status MercuryClient Integration Tests ○ skipped 6 tests Test Suites: 1 passed, 1 total Tests: 6 skipped, 27 passed, 33 total
The console.log line is emitted by the suite itself — an
“Integration Tests Status” block at the foot of the file exists purely to
report why the integration describe was skipped. The totals reconcile: 33
it blocks in the file, six of them inside the conditionally skipped describe,
and TASKS.md 2.2 independently records “27 tests”. The
test script is plain jest, so npm test -- <path>
forwards the path straight through.
Reconciliation & idempotence
Running the cycle twice must not double the ledger. The guarantee is structural —
transactions.mercury_id carries a UNIQUE constraint — and
the live path leans on it directly with onConflictDoUpdate.
// accounts
.onConflictDoUpdate({ target: accounts.mercuryId, /* name, type, balance, lastSyncedAt */ })
// transactions — note what is NOT in the update set
.onConflictDoUpdate({ target: transactions.mercuryId, set: {
amount, description, counterpartyName, syncedAt
// categoryId — preserved, so a classification is never undone
// date — preserved
// direction — preserved
}})
Excluding categoryId from the update set is what makes a re-sync safe: a manual override, or a category the model assigned, survives every subsequent pass. It also means a correction to date or direction upstream will never propagate.
What a freshly synced row looks like
tx.aiCategorized === false
tx.categoryId === null
tx.manuallyOverridden === false
// signed-to-unsigned conversion, asserted on the same rows:
upstream 50000 → amount 50000, direction 'inflow'
upstream -15000 → amount 15000, direction 'outflow'
upstream 0 → amount 0, direction 'inflow'
The zero case carries its own test, titled “should return ‘inflow’ for zero (edge case)”. A fee reversal of exactly zero therefore lands on the inflow side of every aggregate.
Two conventions for the same thing. The live sync.ts writes
direction with an inline ternary, amount ≥ 0 ? 'inflow' :
'outflow'. The shadowed sync/transactions.ts exports a named
calculateDirection() that does the same thing and has thirteen tests. Both
agree on the zero case. Only the second is covered.
Vendor normalisation
Bank counterparty strings are noisy: the same vendor arrives with a run number, a
reference, an order id or a processor prefix appended. Before anything is matched or
learned, the string passes through normalizeVendorName in
src/server/patterns/index.ts.
| Input | Output | Rule exercised |
|---|---|---|
| Case and whitespace | ||
| amazon web services | AMAZON WEB SERVICES | Uppercase |
| Stripe | STRIPE | Trim |
| \t AWS \n | AWS | Trim, including tabs and newlines |
| Google Cloud Platform | GOOGLE CLOUD PLATFORM | Collapse internal runs of whitespace |
| Trailing reference stripping | ||
| GUSTO 121524 | GUSTO | Trailing digits |
| AWS 1234567890 | AWS | Trailing digits, any length |
| ORDER #12345 | ORDER | Trailing # then digits |
| UBER *12345 | UBER | Trailing * then digits |
| LYFT **5678 | LYFT | Repeated * accepted |
| STRIPE TRANSFER 12345 | STRIPE TRANSFER | Multi-word names keep every word |
| Boundaries | ||
| uber *trip #1234 | UBER *TRIP | Only the trailing reference goes; a mid-string * survives |
| McDonald's Restaurant | MCDONALD'S RESTAURANT | Apostrophes are not punctuation to be stripped |
| AMAZON WEB SERVICES | AMAZON WEB SERVICES | Idempotent on already-clean input |
| ' ' | '' | Whitespace-only collapses to empty |
Vendor strings shown here are the test fixtures' own examples — well-known software and transport brands used to illustrate a string-handling rule, not counterparties drawn from the ledger.
The rest of the pattern module
- findPatternMatch(name)
- Exact
ilikefirst, then a substring scan. A hit incrementsmatch_countas a side effect — a read that writes. - createPattern(pattern, categoryId, source)
- Insert or update a row in
category_patterns.sourcedefaults to'manual_rule'. - applyPatternToTransactions(pattern, categoryId)
- Bulk update: sets
category_idandai_categorizedon every transaction whose counterparty matches, excluding rows already flaggedmanually_overridden. Returns the affected row count.
Matching is exact, then substring. A test asserts that the pattern
AMAZON WEB SERVICES does not match the vendor Amazon
— normalisation never shortens a name to its first word. But
findPatternMatch matches in the other direction: a stored pattern appearing
as a substring of a longer incoming name is a hit. The asymmetry lets one learned rule
absorb an entire family of suffixed variants.
Classification
The classifier is built to call the model as rarely as possible. Learned patterns are consulted first, and only vendors that no rule covers are batched off to Claude. Every accepted answer becomes a rule, so the population needing inference shrinks with use.
← Scroll to see the full diagram →
findPatternMatch is free; the right branch costs a model call. Note the
description fallback in the select stage: a wire transfer with no counterparty name is
still classifiable, and the write-back SQL mirrors the same fallback so the match
actually lands. Note also that an unrecognised category name both falls back to
Other and increments the error count — the row is
categorised, but the pass reports it as a failure.
The closed vocabulary
The model is not free to invent a category. The prompt lists twelve labels with
parenthetical examples and demands a bare JSON object; the reply is then looked up
case-insensitively against the twelve seeded names, and a miss resolves to
Other. The vocabulary is fixed at the data layer, not merely requested in
prose.
Who calls it
| Caller | Live? | Note |
|---|---|---|
POST /api/categorize | yes | The */10 Vercel cron, and any manual trigger |
runCategorizationAction() | yes | Server action in src/server/actions/categorize.ts; revalidates /, /transactions, /expenses |
src/server/sync/index.ts | no | Fire-and-forget trigger after a successful sync — in a module nothing imports. See section 2 |
Schedule & authorisation
Two scheduled endpoints on interlocking intervals, declared in vercel.json,
with deliberately different authorisation postures.
{
"crons": [
{ "path": "/api/cron/sync", "schedule": "*/5 * * * *" },
{ "path": "/api/categorize", "schedule": "*/10 * * * *" }
]
}
← Scroll to see the full diagram →
POST /api/categorize runs. The route's own comment explains it —
“Allow either cron secret or no auth for manual triggers”. A wrong
secret is still rejected, so the rule is “absent or correct”. Both paths are
reachable from the open internet: middleware.ts lists
/api/cron(.*) and /api/categorize(.*) as public, so the bearer
check inside /api/cron/sync is the only thing protecting it.
The shared cron helpers
| Export | Status | Body |
|---|---|---|
verifyCronAuth(request) | — | Synchronous. Returns true only when the header equals `Bearer ${process.env.CRON_SECRET}` under a plain === — a byte-wise comparison, not a constant-time one |
unauthorizedResponse() | 401 | { error: 'Unauthorized' } |
cronSuccessResponse(data) | 200 | { success: true, ...data, timestamp } — the payload is spread onto the top level, not nested |
cronErrorResponse(error) | 500 | { error: 'Operation failed', message }; a non-Error throw yields 'Unknown error' |
Both routes decline the generic error label. cronErrorResponse
produces error: 'Operation failed', but the sync route returns
'Sync failed' and the categorise route 'Categorization failed'.
Each builds its own 500 rather than delegating, so that branch of the shared helper is
never taken.
Endpoint contracts
/api/cron/sync |
/api/categorize |
|
|---|---|---|
| Method | GET | POST |
| Other verbs | — | No GET export; Next.js answers 405 |
| Interval | */5 * * * * | */10 * * * * |
| Calls | runIncrementalSync() | categorizeUncategorizedTransactions() |
| Arguments | none | none |
| 200 body | success, accountsSynced, transactionsSynced, newTransactions, timestamp | success, processed, categorized, errors, skippedKnownPatterns, timestamp |
| 401 body | { error: 'Unauthorized' } | { error: 'Unauthorized' } |
| 500 body | { error: 'Sync failed', message } | { error: 'Categorization failed', message } |
The timestamp field is asserted to round-trip: new Date(json.timestamp).toISOString() === json.timestamp. A third cron-adjacent route, GET /api/cron/health, is public and unauthenticated by design and returns { status: 'healthy', timestamp }.