A local-first vision pipeline that looks at a shelf of designer vinyl, names every figure it can prove, prices them, and stages drafts for human review — spending $0 on cloud AI until it genuinely has to.
Every design decision below falls out of these.
Apple's Vision framework finds the objects, an on-device CLIP model matches them, and Apple's on-device foundation model breaks ties. A metered Claude Haiku tier exists in the code but stays dormant without an API key — and a hard ledger caps spend at $5/month even then.
A wrong answer must arrive wearing a low-confidence label. Every accept carries checkable evidence, and the human review step is structural — drafts never publish themselves. Keeping confident-wrong at zero outranks raising the accuracy number.
Reference images come only from compliant, read-only sources. When allvinyls.com's robots.txt forbade image fetching, the answer was a manual drop-folder with automatic labeling — not a workaround.
Builders — human or AI — don't get believed; exit codes, row counts, and diffs do. That discipline caught a leaked accuracy number and an env-var name inside generated HTML before either shipped.
One photo becomes dozens of identified, priced, evidence-backed suggestions. Here's what happens in between.
Apple Vision's saliency request is precise but stingy — at most ~3 boxes per image, no
matter how crowded the shelf. TiledVisionDetector fixes that: slice the
photo into a grid, run saliency per tile, compose the results. The grid adapts to the
photo (derive_adaptive_grid() targets ~420 px tiles): a 2268×4032
portrait becomes a 10×5 grid and 50–100 crops; a 500 px web photo correctly yields a
handful. This was the single biggest accuracy lever found — pre-tiling, a full shelf
produced 14–17 crops and missed most figures.
Overlapping boxes are deduped (IoU > 0.5, higher score wins). Figures straddling a tile seam can duplicate — a known artifact; review is the merge point.
Each crop is embedded with CLIP ViT-B/32 through Apple's Core AI stack —
a float16 .aimodel export driven by a compiled Swift CLI, invoked as a
subprocess. 512-dimensional vectors, ~5 ms per embed, zero Python ML dependencies.
(The PyTorch alternative needed a 14 GiB venv and lost the bake-off.)
Cosine similarity against every reference embedding, aggregated per figure as
the mean of its two best images. This defeats style attractors — a single
real-photo reference among product shots that otherwise absorbs every real-photo query.
Output: top_k_figures(k=5), a ranked candidate list.
Thresholds are never hard-coded — they come from data/tune_report.json,
regenerated from the golden set every training turn. The report carries an
agg field and the pipeline refuses one whose statistic doesn't match
production; a mismatch once produced silently-wrong thresholds, so now it's a hard error.
Each run emits an interactive review.html plus intake.json.
Approved results enter the running Sales Hub through its existing public seam —
POST /api/intake, then PUT /api/items/{id}, localhost only —
arriving as draft items with the full evidence trail in their notes. The
draft → ready → exported state machine keeps a human between the agent and any
marketplace.
Every crop rides the ladder until something accepts it. Each tier costs more than the last; each accept records its method and confidence.
fm CLI: shown the crop and the top-5 candidate thumbnails,
asked to name one with visual evidence (colorway, base stamp). A per-crop failure is a
miss, not a crash — the beta daemon wedges under load and a ~370-call
session must survive it. Names something outside the top-k? Confidence forced low,
nothing chosen.No gradient descent. The model is frozen — what improves is the reference index and the thresholds, one scored batch at a time.
--add-refs or --add-golden, alternating between runs; a
turn ledger records where the alternation stands. A leaked number is never reported as
field accuracy.
| Turn | What happened | Refs | Golden | Top-1 | Caveat |
|---|---|---|---|---|---|
| 0 | Adaptive tiling ships; catalog rebuilt + eBay gap-fill | 693 | 9 | 77.8% | honest baseline |
| 1 | Runs 1–6 scored (407 crops), both flags applied | 713 | 29 | 93.1% | leaked |
| 2 | Runs 7–9 scored; alternation begins | 730 | 38 | 92.1% | partially leaked |
| 3 | The purge: 371 operator-marked bad eBay refs quarantined | 365 | 44 | 88.6% | honest drop — index now trustworthy |
| 4 | Runs 19–22 scored; runs 23–46 produced (web photos) | 370 | 45 | — | thr 0.8274 · awaiting audit |
The purge is the day's most important event: the eBay enrichment that once supplied most of the index turned out to be ~80% junk — wrong figures on real listings. The accuracy number fell when it left; the accuracy meaning rose. The binding constraint now is coverage: 193 figures have no reference image at all, and the cure is exactly this loop plus the drop-folder.
Everything runs from the repo root with one environment block. The pinned agent venv is the only correct interpreter — the app's venv deliberately lacks the agent's dependencies, and vice versa.
# environment — every session, verbatim
cd ~/Developer/projects/listing-tool
export PYTHONPATH=$PWD \
AGENT_COREAI_CLI=$PWD/agent/probes/bakeoff/coreai/coreai_bakeoff/.build/out/Products/Release/coreai_clip_cli \
AGENT_COREAI_MODEL=/Volumes/1TBEDRIVE/Workingspace/coreai-models/exports/clip-vit-base-patch32_float16_static.aimodel \
AGENT_VISION_DETECT_CLI=$PWD/agent/probes/vision_detect_cli
PY=/Volumes/1TBEDRIVE/Workingspace/agent-venv/bin/pythonagent.intake.pipeline — photo in, suggestions outThe main event. Writes crops, DB rows, and suggestions.json
under data/catalog/intake/<run_id>/.
$PY -m agent.intake.pipeline <photo> --tune-report data/tune_report.json \
--db-path data/catalog.db --run-notes "context"
# --grid 8x7 / --target-tile-px N override adaptive tiling (never 0)
# --dry-run prints the crop/tier plan without any model callsagent.handoff.run — review artifacts, optional pushGenerates the interactive review.html + intake.json.
--push sends all approved results of that run into the live app as drafts.
$PY -m agent.handoff.run <run_id> [--push] [--app-url http://127.0.0.1:8000]agent.handoff.apply_decisions — the flywheel stepApplies a scored decisions JSON: overrides become manual/high, rejects clear the choice, confirms stand. All-or-nothing validation, atomic replacement. One growth flag per run, alternating.
$PY -m agent.handoff.apply_decisions <run_id> <decisions.json> --add-refs # or --add-golden
# then verify counts grew: user refs + golden must both be ≥ last turnagent.matching.tune — thresholds from the golden setReplays every labeled photo in data/golden_ids/ against the
index. Copy the numbers into data/tune_report.json with
"agg": "mean_top2" — the pipeline rejects a mismatched statistic.
$PY -m agent.matching.tune --golden data/golden_ids --backend coreai --db data/catalog.dbagent.probes.build_audit_site — remote audit packageBuilds a self-contained static site (all images inlined) for remote scoring —
always with --dataset so both audit surfaces ship.
$PY -m agent.probes.build_audit_site --runs 23,24,… --dataset --out data/audit_site_vN
# then: grep -rqE "/Users|/Volumes|ANTHROPIC|sk-|Bearer|API_KEY" out/ → must find nothingagent.probes.apply_corrections_purge — dataset cleanupConsumes the dataset page's corrections export. Quarantine, not delete: files move aside, embeddings and rows go. Dry-run by default; the destructive step needs operator approval.
$PY -m agent.probes.apply_corrections_purge <corrections.json> [--apply]agent.probes.import_allvinyls_drop — compliant catalog growthallvinyls.com forbids automated image fetches, so the operator saves images manually (keep original filenames) into the drop folder. The importer auto-labels each file by parsing the cached series-page HTML — filename → figure, no manual tagging.
$PY -m agent.probes.import_allvinyls_drop <drop-dir> [--apply] # dry-run firstScoring, packaged for anyone with a browser — deploy the folder to Cloudflare
Pages or open it straight from file://.
Every crop with its top-5 candidates as radio buttons, an approve checkbox, a free
note, and three label fields — figure, series, artist — for items the
catalog doesn't know yet. Export downloads a decisions JSON that
apply_decisions ingests unchanged (labels pack into the note as
label: figure=…|series=…|artist=…).
Every reference image as a thumbnail with a "bad" checkbox, grouped by figure with source labels, plus the "needs images" list of figures with no reference at all. Export produces the corrections format the purge tool consumes.
ANTHROPIC_API_KEY from a ladder-evidence line — and the final gate is a manual
grep for /Users, /Volumes, and key patterns, run pipe-free so the
exit code is real.
| Path | What it is |
|---|---|
data/catalog.db | SQLite: series, figures, reference images, embeddings, intake runs/results. Gitignored but persistent — never in tmp. |
data/catalog/images/<fid>/ | Reference files per figure; confirmed crops arrive as user_crop_<run>_<result>.jpg. |
data/golden_ids/ | The measurement set: labeled real photos named <figure_id>__qN.jpg. |
data/catalog/intake/<run>/ | Per-run crops, suggestions.json, review.html, intake.json. |
data/tune_report.json | The live thresholds: {n, threshold, mean_margin, agg}. |
data/catalog/images_quarantine/ | Purged refs (reversible), named <fid>__<original>. |
agent/.env | Secrets, env-only, never committed: eBay API creds, dormant Anthropic key slot. |
/Volumes/1TBEDRIVE/Workingspace/ | The heavy stuff: pinned agent venv, Core AI model exports, drop folders. |
Exit codes die in pipes — cmd > out 2>&1; echo $?, never
cmd | tail; this shipped a false "tests pass" twice.
/private/tmp is wiped on restart; durable state lives only in repo
data/ or the external drive.
The fm daemon wedges after heavy batches — restart the Mac; the ladder
degrades safely meanwhile.
exFAT worktrees show every file modified until
core.fileMode false.
Small web photos (~500 px) legitimately yield 2–8 crops — adaptive tiling working,
not breaking.
negative Dark-background ref
augmentation — compositing refs onto black to match dark shelves. No gain,
cleanly reverted.
negative Query-side dark matting —
actively hurt dark figures.
negative Self-consistency quarantine @
0.72 — caught nothing; the "attractors" were genuinely correct photos. The real
fix was mean-top-2 aggregation.
| Metric | Value |
|---|---|
| Reference index | 370 images — every one operator-vetted or original-dataset (48 from confirmed crops) |
| Golden set | 45 labeled real photos |
| Live thresholds | 0.8274 accept · 0.0815 margin · mean-top-2 |
| Confident-wrong, all-time | 0 |
| Figures needing images | 193 — the binding constraint |
| Awaiting | decisions for runs 23–46 · next dataset-audit corrections · ~75 unprocessed photos |
The path from here is more of the same loop: score the outstanding runs, refill the 193 imageless figures via the drop-folder, and keep alternating refs and golden until the tune numbers turn boringly stable. The machinery is done — from now on, the work is feeding it.