Measured results, misses included
40 seeded synthetic bills · mode: rules-only (no model) · generated Sat, 14 Nov 2026 12:30:51 GMT · 20 ms total. Reproduce with npm run eval.
Labeled PII leaked
0 / 440
Free-text PII leaked (rules-only)
67 / 68
Clean bills falsely flagged
0 / 11
Disputable $ found
$13,662
Billing-error rules
| Rule | TP | FP | FN | Precision | Recall |
|---|---|---|---|---|---|
| DUPLICATE | 10 | 0 | 0 | 100% | 100% |
| MATH | 6 | 0 | 0 | 100% | 100% |
| TOTAL | 7 | 0 | 0 | 100% | 100% |
| UNBUNDLE | 10 | 0 | 0 | 100% | 100% |
| POST_DISCHARGE | 6 | 0 | 0 | 100% | 100% |
| PRICE_OUTLIER | 8 | 0 | 0 | 100% | 100% |
| DOUBLE_EM | 7 | 0 | 0 | 100% | 100% |
Every miss (67)
The free-text leaks are why BillShield adds an on-device Gemini Nano pass: the regex layer alone misses spaced phone numbers, spelled-out birthdays and first names in notes. Model-on numbers will be added after measuring in Chrome with Nano.
| bill-100 | PII leak (free-text, rules-only) | born September ##, #### |
| bill-101 | PII leak (free-text, rules-only) | born October #, #### |
| bill-102 | PII leak (free-text, rules-only) | Daniel |
| bill-102 | PII leak (free-text, rules-only) | ### ### #### |
| bill-102 | PII leak (free-text, rules-only) | born October ##, #### |
| bill-103 | PII leak (free-text, rules-only) | Lucas |
| bill-103 | PII leak (free-text, rules-only) | ### ### #### |
| bill-103 | PII leak (free-text, rules-only) | born October #, #### |
| bill-104 | PII leak (free-text, rules-only) | born October ##, #### |
| bill-105 | PII leak (free-text, rules-only) | Sofia |
| bill-105 | PII leak (free-text, rules-only) | ### ### #### |
| bill-106 | PII leak (free-text, rules-only) | Aisha |
| bill-106 | PII leak (free-text, rules-only) | ### ### #### |
| bill-107 | PII leak (free-text, rules-only) | Daniel |
| bill-107 | PII leak (free-text, rules-only) | ### ### #### |
| bill-108 | PII leak (free-text, rules-only) | Lucas |
| bill-108 | PII leak (free-text, rules-only) | ### ### #### |
| bill-108 | PII leak (free-text, rules-only) | born November ##, #### |
| bill-109 | PII leak (free-text, rules-only) | ### ### #### |
| bill-109 | PII leak (free-text, rules-only) | born November #, #### |
| bill-110 | PII leak (free-text, rules-only) | born November ##, #### |
| bill-112 | PII leak (free-text, rules-only) | Daniel |
| bill-112 | PII leak (free-text, rules-only) | ### ### #### |
| bill-112 | PII leak (free-text, rules-only) | born December ##, #### |
| bill-113 | PII leak (free-text, rules-only) | Fatima |
| bill-113 | PII leak (free-text, rules-only) | ### ### #### |
| bill-113 | PII leak (free-text, rules-only) | born December ##, #### |
| bill-114 | PII leak (free-text, rules-only) | Elena |
| bill-114 | PII leak (free-text, rules-only) | ### ### #### |
| bill-117 | PII leak (free-text, rules-only) | Priya |
| bill-117 | PII leak (free-text, rules-only) | ### ### #### |
| bill-117 | PII leak (free-text, rules-only) | born January ##, #### |
| bill-118 | PII leak (free-text, rules-only) | Fatima |
| bill-118 | PII leak (free-text, rules-only) | ### ### #### |
| bill-118 | PII leak (free-text, rules-only) | born January ##, #### |
| bill-120 | PII leak (free-text, rules-only) | born January ##, #### |
| bill-121 | PII leak (free-text, rules-only) | Aisha |
| bill-121 | PII leak (free-text, rules-only) | ### ### #### |
| bill-122 | PII leak (free-text, rules-only) | Priya |
| bill-122 | PII leak (free-text, rules-only) | ### ### #### |
| bill-123 | PII leak (free-text, rules-only) | born February ##, #### |
| bill-125 | PII leak (free-text, rules-only) | born February ##, #### |
| bill-126 | PII leak (free-text, rules-only) | Wei |
| bill-126 | PII leak (free-text, rules-only) | ### ### #### |
| bill-127 | PII leak (free-text, rules-only) | born February ##, #### |
| bill-128 | PII leak (free-text, rules-only) | Noah |
| bill-128 | PII leak (free-text, rules-only) | ### ### #### |
| bill-128 | PII leak (free-text, rules-only) | born March #, #### |
| bill-129 | PII leak (free-text, rules-only) | born March ##, #### |
| bill-132 | PII leak (free-text, rules-only) | Priya |
| bill-132 | PII leak (free-text, rules-only) | ### ### #### |
| bill-133 | PII leak (free-text, rules-only) | Noah |
| bill-133 | PII leak (free-text, rules-only) | ### ### #### |
| bill-133 | PII leak (free-text, rules-only) | born April ##, #### |
| bill-134 | PII leak (free-text, rules-only) | Kwame |
| bill-134 | PII leak (free-text, rules-only) | ### ### #### |
| bill-134 | PII leak (free-text, rules-only) | born April #, #### |
| bill-136 | PII leak (free-text, rules-only) | Wei |
| bill-136 | PII leak (free-text, rules-only) | ### ### #### |
| bill-136 | PII leak (free-text, rules-only) | born April #, #### |
| bill-137 | PII leak (free-text, rules-only) | Lucas |
| bill-137 | PII leak (free-text, rules-only) | ### ### #### |
| bill-138 | PII leak (free-text, rules-only) | Noah |
| bill-138 | PII leak (free-text, rules-only) | ### ### #### |
| bill-139 | PII leak (free-text, rules-only) | Sofia |
| bill-139 | PII leak (free-text, rules-only) | ### ### #### |
| bill-139 | PII leak (free-text, rules-only) | born May ##, #### |
Cloud letter path (Firebase AI Logic, live site)
Each letter drafted by the cloud model is checked by deterministic code: every dollar amount and billing code must trace to a finding, no new placeholders, no raw identifiers. Anything that fails falls back to the template. Measured with headless Chrome against the live deployment.
Bills
12
Cloud letters accepted
12 / 12
Rejected by fact checker
0
Redaction guard passed
12 / 12
gemini-3.5-flash-lite via Firebase AI Logic · headless Chrome (no Gemini Nano) → cloud path · 2026-11-14T12:30:06.783Z
| bill-100 | in_cloud | PASSED: 8 amounts, 6 codes all traced to findings | |
| bill-101 | in_cloud | PASSED: 7 amounts, 3 codes all traced to findings | |
| bill-102 | in_cloud | PASSED: 6 amounts, 7 codes all traced to findings | |
| bill-103 | in_cloud | PASSED: 7 amounts, 3 codes all traced to findings | |
| bill-104 | in_cloud | PASSED: 9 amounts, 3 codes all traced to findings | |
| bill-105 | in_cloud | PASSED: 7 amounts, 7 codes all traced to findings | |
| bill-106 | in_cloud | PASSED: 9 amounts, 2 codes all traced to findings | |
| bill-118 | in_cloud | PASSED: 2 amounts, 1 codes all traced to findings | |
| bill-119 | in_cloud | PASSED: 2 amounts, 6 codes all traced to findings | |
| bill-120 | in_cloud | PASSED: 4 amounts, 0 codes all traced to findings | |
| bill-121 | in_cloud | PASSED: 3 amounts, 1 codes all traced to findings | |
| bill-122 | in_cloud | PASSED: 2 amounts, 2 codes all traced to findings |
Run history (what the checker caught)
- v1 (2026-11-14T12:23:25Z): 4 accepted, 4 rejected, 4 call failures. Model invented placeholders ([DATE_1], [ADDRESS_2]) in 4 letters; 1 also cited $15.00. The checker rejected them and the template was used. 4 calls failed with 401 'Firebase App Check token is invalid' (App Check enforcement change still propagating; web app not registered in App Check).
- v2 (2026-11-14T12:26:22Z): 4 accepted, 2 rejected, 6 call failures. Prompt tightened (no new placeholders, only quote amounts verbatim): 0 invented placeholders. The 2 rejections ($15.00) were a CHECKER false positive: the finding text says 'a typical $15 charge' without cents. Fixed the checker to accept whole-dollar amounts. 6 calls still failed with the same intermittent 401.
- v3 (2026-11-14T12:30:06Z): 12 accepted, 0 rejected, 0 call failures. After the checker fix and a one-retry policy (the retry sends no App Check header): 12/12 letters drafted in the cloud by gemini-3.5-flash-lite, 12/12 passed the fact check, 12/12 redaction-guard passes. Some calls needed the retry after a transient 401 'Firebase App Check token is invalid' while the enforcement change propagated.
Notes
- Synthetic, seeded bills (fictional people). Ground truth = planted errors + every PII string written into the bill.
- 'Free-text' PII is deliberately written in formats the regex layer does not cover (spaced phone numbers, spelled-out birthdays, spouse first names). These misses are what the on-device Gemini Nano pass is for; model-on numbers will be published separately once measured in Chrome.
- Eval history: the first run (v0.1) produced 10 false positives: 8 DUPLICATE flags on a legit same-code venipuncture decoy (a generator bug that put both draws on the same day) and 2 DOUBLE_EM flags caused by counting an exact duplicate E/M line twice (an engine bug, now fixed: exact duplicates are reported once, under DUPLICATE).
- The leak gate re-checks every outbound payload; gateBlocks counts payloads it would refuse to send.