Record linkage with active learning
The demo selects record pairs for labeling by expected information gain. The resulting labels are used to fit a calibrated model that matches record pairs for deduplication or record linkage.
Early on you’ll see sim (raw schema-aware similarity). After a few labels, you’ll see prob (calibrated
match probability) and the queried pairs concentrate near p≈0.5.
- Upload CSV(s)
- Label 10–30 pairs
- Run full scoring + download results
How it works
- Treat every compatible matching-name CSV column as potential matching evidence.
- Compare aligned text fields field-by-field, and use scaled numeric differences for numeric fields.
- Screen candidate pairs with a schema-aware similarity score before fitting the active learner.
- Compress the resulting features with PCA into a small embedding (principal components).
- Fit ridge logistic regression on PCs to predict match probability.
- Select the next pairs by expected reduction in model uncertainty.
- Query strategy: start with diverse pairs, then pick batches that maximize expected information gain.
As labels accumulate, the backend can increase capacity by using more PCs.
This page calls a hosted backend API. Uploads are processed for your session and used only to fit the in-session matcher (not for training outside your run). Data may be cached temporarily to generate outputs and is not intended to be retained as a dataset.
The public backend runs on CPU only and may be rate-limited. For private-access or higher-throughput, request joe@josephsmiller.com.
Upload
CSV format (hosted demo): upload one CSV for deduplication or left/right CSVs for linkage. Matching-name columns
are treated as candidate matching evidence, and a text/name-like column is used as the display string.
Use standard CSV quoting for commas inside a field. If you are unsure about the format, download one of the sample CSVs first and match that structure.
Sample datasets are optional — you can upload your own CSVs instead (subject to demo limits enforced by the backend).
Label
Suggestions only apply to high-confidence probabilities. Suggested labels are highlighted until you review the pair; uncertain pairs are left blank.
Compared fields: this hosted backend infers compatible fields from your CSVs and compares multiple columns field-by-field when they are available.
Keyboard shortcuts
j or ↓: next • k or ↑: previous • a: match • s: skip • d: no-match •
x: clear • Enter: submit • Shift+N: next batch
Results
Predictions = scored candidate pairs (with probabilities once you’ve labeled a few pairs). Matches = linkage-mode pairs above the backend match threshold. Clusters are generated only for dedup mode (linkage mode returns matches instead).
- predictions: review scores and choose a cutoff for your workflow
- matches: linkage-mode “accept list” (mutual best matches above the backend threshold)
- clusters: dedup-mode group IDs produced from clustering the scored similarities
Metrics update as you label batches. Use “Run full scoring” to generate downloadable outputs.