Facilitator 502 → clean 402: fix status

2026-09-07 ~06:00 UTC. Branch fix/cdp-verify-invalid-verdict in sudzikcoin/suverse-pay, built and tested in an isolated worktree (/home/govhub/suverse-pay-wt). DEPLOYED 2026-09-07 06:06 UTC on owner go. Status: LIVE, verified, no rollback. Root-cause report: funnel-report-20260907.html.

1. The two code fixes

Commit	What	Files
`84945e6`	FIX 1 — CDP 4xx verdict → proper 402. `httpJson` gains an opt-in `acceptStatus` (a JSON 4xx body is returned as data instead of thrown). The CDP adapter's `verify()` accepts 4xx and, when the body is a verdict with `isValid:false`, returns `valid:false` with the mapped code and CDP's own text; any other 4xx body re-throws with exactly the code and message it had before. `invalid_payload` is now mapped (→ `invalid_authorization`), refined to `insufficient_funds` when the message shows the transfer simulation reverted (the 0x8a1A… body). Settle path untouched.	packages/provider-sdk/src/http-json.ts · packages/adapters/coinbase-cdp/src/{adapter,error-map}.ts + tests
`f474c8c`	FIX 2 — nested error read, logged text, no retry on verdicts. `readFacilitatorError()` reads both the flat `{errorCode,errorMessage}` and our facilitator's nested `{error:{code,message,details}}`. Every transient / exhausted / rejected log line now carries the (truncated) provider message. Payment-verdict codes (`insufficient_funds`, `invalid_signature`, `invalid_authorization`, `invalid_payload`, `nonce_already_used`, `expired_authorization`, `insufficient_grant`, `unsupported_scheme`, `route_unsupported`) are never retried and always yield a 402 with the reason, whatever HTTP status wrapped them.	packages/x402-server-node/src/core.ts + tests

 packages/adapters/coinbase-cdp/src/adapter.test.ts |  62 ++++++++++++
 packages/adapters/coinbase-cdp/src/adapter.ts      |  50 +++++++++-
 .../adapters/coinbase-cdp/src/error-map.test.ts    |  40 +++++++-
 packages/adapters/coinbase-cdp/src/error-map.ts    |  45 +++++++++
 packages/provider-sdk/src/http-json.test.ts        |  34 +++++++
 packages/provider-sdk/src/http-json.ts             |  34 +++++++
 .../x402-server-node/src/__tests__/core.test.ts    | 101 +++++++++++++++++++
 packages/x402-server-node/src/core.ts              | 110 +++++++++++++++++----
 8 files changed, 452 insertions(+), 24 deletions(-)

Branch base: 628ba93 (feat/no-wallet-hint = what is live on the proxy since 07:10 CET today; api is live at main + c92e960, both contained in this base). Commits carry no trailer, per the repo's convention. Not pushed.

2. Build / test status (worktree, turbo cache disabled)

Step	Result
`pnpm turbo run build --force` (whole monorepo)	25/25 tasks OK, 0 cached, 1m48s (post-rebase run)
`pnpm turbo run typecheck test --force` (whole monorepo, first run)	63/68 tasks OK; the 5 failures are `apps/proxy` tests `carrier-risk-verdict` (4) and `road-conditions` (1) — pre-existing: identical 5 failures on the untouched live tree (503 vs 404 from the FMCSA mirror preflight). Every other package green.
Changed packages + api, post-rebase	x402-server 52/52 · provider-sdk 42/42 · adapter-coinbase-cdp 67/67 · api 83/83 · proxy handler.test 32/32
New tests	13 added: httpJson acceptStatus ×3, mapCdpVerifyRejection ×3, adapter 4xx-verdict ×5, middleware nested-error / verdict-no-retry / 200-verdict ×3
Artifact check (grep the built dist, not the exit code)	`provider-sdk/dist/http-json.js` acceptStatus=1 · `coinbase-cdp/dist/adapter.js` mapCdpVerifyRejection=2 · `x402-server-node/dist/core.js` readFacilitatorError=2, PAYMENT_VERDICT_CODES=2 · `apps/proxy/dist/handler.js` no_wallet=4 (live feature preserved). apps resolve these packages through workspace symlinks to `dist/`, not bundled.

3. Sample: invalid payment, before vs after

Before (live now, measured 2026-09-07 05:49 UTC with a fresh random wallet holding 0 USDC, X-PAYMENT for $0.001 on bazaar-test):

HTTP 502 in 1146 ms
body.error = "facilitator verify returned HTTP 502"        ← 3 facilitator round-trips, no reason

After (new dist, replaying the exact CDP HTTP 400 body captured for payer 0x8a1A…):

CDP adapter        → {"valid":false,"providerId":"coinbase-cdp","payer":"0x8a1A037b…","errorCode":"insufficient_funds",
                       "errorMessage":"contract call failed: unable to call contract: execution reverted"}
facilitator /verify → HTTP 200 {"isValid":false,"invalidReason":"insufficient_funds","payer":"0x8a1A037b…"}
proxy → buyer      → HTTP 402, ONE facilitator call, body:
{"x402Version":2,
 "resource":{"url":"https://proxy.suverse.io/v1/proxy/reskey_1166628d/bazaar-test","description":"bazaar-test","mimeType":"application/json"},
 "accepts":[{"scheme":"exact","network":"eip155:8453","asset":"0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
             "payTo":"0x260fbe1ec46968ee02e5b972507d7bb7f09f82b0","amount":"1000","maxTimeoutSeconds":60,"extra":{"name":"USD Coin","version":"2"}}],
 "error":"insufficient_funds"}
proxy_request_logs → outcome=settle_failed, error_code=insufficient_funds (was facilitator_error)

Defence in depth (FIX 2 alone, e.g. an old facilitator still answering 502): the buyer still gets the 502 for an invalid_request-coded failure (kept retryable on purpose, Jun-2026 CDP outage class), but the log line now reads

… code=invalid_request status=502 message=POST https://api.cdp.coinbase.com/…/verify -> HTTP 400: {"invalidMessage":"contract call failed: … execution reverted","invalidReason":"invalid_payload",…}

instead of a bare status. A verdict code wrapped in a 5xx is answered 402 in one call.

4. Valid payments: unchanged

Check (old live dist vs new dist, same mocked CDP replies)	Result
CDP adapter `verify()` output for `{"isValid":true,"payer":…}`	identical `{"valid":true,"providerId":"coinbase-cdp","payer":"0x3869…"}`
Middleware outbound /verify + /settle request envelopes (URL, method, headers, body)	byte-identical
Middleware accepted receipt (payer, network, asset, amount, txHash)	byte-identical
Code path	2xx branch of `httpJson` untouched; `acceptStatus` is consulted only after the 2xx return. Adapter 200-verdict path unchanged (one console.log line now prefixes the HTTP status). Middleware `response.ok` path unchanged. Settle path unchanged.

5. FIX 3 — QA wallet top-up (you send, I don't)

Wallet	Address	Balance now	Burn	Send
QA bot Base (internal_wallets #7, Base swap-quote QA from the AgentOS host)	`0x0145ee0B440300928291668eDC5557f4B0779087`	0.000455 USDC (+0.00358 ETH, gas is not needed: facilitator pays)	~8 settles/day × $0.001 = ~$0.24/month (measured: 7 settles in the 18 h it still worked; 105 attempts blocked since)	5 USDC on Base (eip155:8453), native USDC `0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913` ≈ 20 months. 10 USDC if you want to forget about it.
QA bot Solana (internal_wallets #6) — heads-up, same cliff in ~20 days	`GEytKjbGUTnBH2M55sRNMibim2LgLMamHBRnXXofdDQk`	0.163992 USDC	111 settles / 14 d = ~8/day = ~$0.24/month	5 USDC (SPL USDC `EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v`) whenever convenient

Verified on-chain 2026-09-07 ~05:45 UTC via base-rpc.publicnode.com and api.mainnet-beta.solana.com. Minimum that unblocks Base QA immediately: anything ≥ 0.001 USDC; 5 USDC is the sensible amount.

6. Deploy + rollback (deploy RUN 06:04–06:07 UTC; rollback commands still valid)

# Pre-deploy snapshot ALREADY TAKEN (662 files):
#   /home/govhub/backups/dist-pre-cdp-verify-fix-20260907-0747.tar.gz   (live tree HEAD = 628ba93)

# DEPLOY (main tree; services load /home/govhub/suverse-pay/{apps/*,packages/*}/dist via workspace symlinks)
cd /home/govhub/suverse-pay
git status --short            # expect only the 4 known untracked files
git checkout fix/cdp-verify-invalid-verdict          # fast-forward from 628ba93 (+2 commits)
pnpm turbo run build --force                           # full, cache bypassed
grep -c acceptStatus packages/provider-sdk/dist/http-json.js            # expect 1
grep -c mapCdpVerifyRejection packages/adapters/coinbase-cdp/dist/adapter.js  # expect 2
grep -c readFacilitatorError packages/x402-server-node/dist/core.js     # expect 2
grep -c no_wallet apps/proxy/dist/handler.js                            # expect 4 (live feature kept)
kill -TERM $(systemctl show -p MainPID --value suverse-pay-api.service)    # Restart=always brings it back (~5 s)
journalctl -u suverse-pay-api.service -n 20 --no-pager | grep -i listening
kill -TERM $(systemctl show -p MainPID --value suverse-pay-proxy.service)
journalctl -u suverse-pay-proxy.service -n 20 --no-pager | grep -i listening
# order matters: api (facilitator) first, then proxy; between the two restarts the old proxy talks to the new facilitator, which is
# a strictly-better combination (200 isValid:false → old middleware already maps it to 402).

# LIVE VERIFICATION (scripts ready in the session scratchpad)
node live-check.mjs invalid   # random empty wallet → expect HTTP 402, body.error "insufficient_funds", 1 facilitator call
node live-check.mjs verify    # read-only /facilitator/verify with 0x3869…, $0.001 → expect 200 {"isValid":true}
node live-check.mjs settle    # REAL $0.001 from 0x3869… to bazaar-test → expect 200 + x-payment-response, row in facilitator_payments
journalctl -u suverse-pay-proxy.service --since "-10 min" | grep "facilitator verify rejected"   # nested text now in logs
# plus: watch the next 0x8a1A… attempt (~every 4 h) turn into settle_failed/insufficient_funds instead of facilitator_error

# ROLLBACK (≤ 1 min, no rebuild)
cd /home/govhub/suverse-pay
tar xzf /home/govhub/backups/dist-pre-cdp-verify-fix-20260907-0747.tar.gz      # restores the 5 dist dirs
git checkout feat/no-wallet-hint                                              # source back to 628ba93
kill -TERM $(systemctl show -p MainPID --value suverse-pay-api.service); sleep 6
kill -TERM $(systemctl show -p MainPID --value suverse-pay-proxy.service)
node live-check.mjs verify                                                    # confirm valid path again

Trigger for rollback: any change in the valid path — live-check.mjs verify not returning isValid:true, or live-check.mjs settle not settling, or a settled row missing from facilitator_payments.

7. Live verification (2026-09-07 06:06–06:08 UTC)

Step	Result
Deploy	live tree `git checkout fix/cdp-verify-invalid-verdict` (f474c8c), `pnpm turbo run build --force` 25/25 · 0 cached · 2m04s; dist grep in the live tree: acceptStatus=1, mapCdpVerifyRejection=2, readFacilitatorError=2, PAYMENT_VERDICT_CODES=2, no_wallet=4. First attempt was refused by git because the branch was still attached to the worktree, and a pipe hid the failure, so one build ran on the old commit (dist byte-identical, nothing restarted). Worktree removed, second attempt clean.
Restarts	suverse-pay-api pid 681562 → 3146172, `/facilitator/health` ok; then suverse-pay-proxy pid 3090530 → 3146261, "proxy listening", 402 probe ok. Restart gap per service ≈ 5 s.
Invalid payment (fresh random wallet 0x8a6115…, 0 USDC, X-PAYMENT $0.001 on bazaar-test)	HTTP 402 in 673 ms, body `"error":"insufficient_funds"`, challenge body otherwise normal. `proxy_request_logs`: `settle_failed / insufficient_funds` (the same call 20 min earlier, pre-deploy: HTTP 502 / `facilitator_error` in 1146 ms after 3 facilitator round-trips).
Valid verify (read-only, 0x3869…, $0.001)	200 `{"isValid":true,"payer":"0x3869dE75…"}`
Valid settle (real $0.001, 0x3869… → bazaar-test)	HTTP 200 in 1386 ms, upstream body served, `x-payment-response` = `{"success":true,"transaction":"0x7863bb54e156116c77d1f35d7000c78e72b2f7fa5e693d4710642d54de1f7c72","network":"eip155:8453","payer":"0x3869dE75…","amount":"1000"}`; `facilitator_payments` fpay_01M1X7MZHZ8WR567T0E8RWGPW9 settled; `proxy_request_logs` settled with the tx hash.
Logs	api journal now records the provider text for the invalid case: `[CDP-VERIFY-BODY] HTTP 400 {"invalidMessage":"contract call failed: unable to call contract: execution reverted","invalidReason":"invalid_payload","isValid":false,"payer":"0x8a6115…"}`. The proxy-side nested-error line ("facilitator verify rejected … message=…") fires only when the facilitator itself answers non-2xx; with FIX 1 live that path was not exercised (the verdict now arrives as 200), it is covered by the unit tests and the dist harness.
Regressions	None seen. No rollback. External attempts after deploy so far: none yet (the 0x8a1A… bot is due ~09:00 UTC; expect `settle_failed / insufficient_funds` instead of `facilitator_error`).
Noticed, pre-existing, not caused by this deploy	api startup logs `ERROR admin api_key row 'apikey_admin_default' hash does not match ADMIN_API_KEY env` — a startup sanity check in `apps/api/src/index.ts` that only logs and continues; the DB row and the env value have drifted (the service had not restarted since 07-28, journald starts 08-12, so the earlier startup line is not retrievable). Payment path unaffected (verified above). Fix when convenient: `pnpm db:bootstrap` or align the env key.

Rollback remains available: /home/govhub/backups/dist-pre-cdp-verify-fix-20260907-0747.tar.gz + git checkout feat/no-wallet-hint + the two restarts (section 6).