Smart Scale
Camera-based produce recognition for retail scales.
- Python
- MNN
- Electron
- FastAPI
- OpenCV
- SQLite
- .NET 8
- PostgreSQL + pgvector
- Next.js
- Docker
Closed source · case study
The problem
At a produce scale, the cashier has to find the right item in a long list, for every customer. The scales shipped with the vendor's own app; this replaces it with one that recognizes the produce and learns from every confirmed sale.
It has to keep selling with no network, never hold up a sale when it's unsure, and cope with produce that looks different from shop to shop and season to season, without retraining a model each time.
My role
I pushed for the recognition approach and owned its design.
- The case for embeddings over a local classifier or a paid vision API
- Reusing the scale's own embedding model instead of shipping a new one
- The learning loop: a confirmed pick becomes a reference for its category
- How references move between scales: through the Portal API, gated by approval and pipeline signature
Architecture
ALSO IN THE SYSTEM
- A Python agent on each scale owns the weighing scale, camera, model, index and label printer; the Electron kiosk only draws the screen
- Only 1,024-number fingerprints travel between scales, never photos; ranking happens on the scale, and Postgres with pgvector only stores them
- Two pools per category, own and shared, with the oldest dropped first: a mislabelled photo is the least like the rest, so the rule that drops redundant references could never drop it
- Labels are drawn as bitmaps because the printer's text mode can't shape Arabic, and carry an EAN-13 with product code and weight
- Licensing: one sign-in per shop day, then the scale sells offline all day; clock rollback is detected
Key decisions
Embeddings, not a classifier
- CHOSE
- Embed each photo and rank it against stored references for every category.
- REJECTED
- Training a local classifier, or paying for a vision API.
- WHY
- Teaching the system a new category means adding references, not running a training job. And everything runs on the scale, so the till keeps working offline.
Keep the scale's own model, after measuring
- CHOSE
- The MobileNetV2-based embedding model that shipped with the scale, run on-device.
- REJECTED
- Six open models: CLIP, SigLIP, ResNet-50, EfficientNet-B0, MobileNetV2 and MobileNetV3.
- WHY
- On the shop's own produce photos it ranked first once a category had three references, and it was already on the hardware. The same benchmark showed its weak spot: with a single reference, two open models did better.
A shortcut, never a gate
- CHOSE
- Suggest up to three categories, and only let one lead when it clearly wins. The cashier picks the exact product, and only labels a person confirmed are learned.
- REJECTED
- Auto-selecting the top match.
- WHY
- A wrong guess costs a tap, not a wrong sale. And a corrected pick is worth more than any other reference: it's the case the model got wrong, labelled by someone looking at the item.
Share fingerprints, gate what spreads
- CHOSE
- A new category needs approval with real photos. A scale only accepts references made by the same pipeline, checked by a signature of model, size, crop and normalisation.
- REJECTED
- Every scale learning alone, which was the first design, or sharing everything unchecked.
- WHY
- One mislabeled reference shouldn't spread to every shop, and a changed camera or crop setting must be detected instead of silently ruining every score. A QA pass caught exactly that: one setting change had cut off every approved reference.
Results
Live
with the launch client, tenant rollout starting
1st of 7
embedding models benchmarked on the shop's own produce, at three references
1,274
automated tests on the scale agent
1,153
automated tests on the .NET API
Recognition quality is measured live: the tenant dashboard shows 7-day top-3 accuracy against the previous period, and the pairs the camera confuses.
What I'd do next
- 01
Shrink the on-scale index's memory, so shops with 150+ categories fit on the same hardware.
- 02
Record which shop each shared reference came from, so a bad one can be traced and removed.
- 03
Keep source photos, so a pipeline change re-embeds references instead of every shop teaching from scratch.