Every CRM we audit has the same disease. "Acme Corp," "ACME Corporation," "Acme, Inc.," and "acme.io" are four accounts with four owners, four lead scores, and four slices of the same buying committee. Your routing sends the second contact to the wrong rep, your ABM list counts one company as four, and your attribution splits a single deal across records that never talk to each other.
The usual fix is a dedupe tool with fuzzy-match rules, and it fails in a predictable way. Tight thresholds miss "International Business Machines" versus "IBM." Loose thresholds merge "Delta Air Lines" with "Delta Faucet." Teams respond by turning auto-merge off, and the duplicate count climbs again.
Our position: entity resolution is now an AI Tools problem, not a rules problem. The right pattern is a three-stage pipeline where cheap deterministic rules handle the obvious cases, embeddings find candidate pairs that string matching cannot see, and an LLM adjudicates only the ambiguous middle with a structured verdict. Nothing merges without an audit trail.
Stage One: Block Before You Compare
Comparing every record to every other record does not scale. A 200,000 account CRM has roughly 20 billion possible pairs. Blocking narrows that to a few hundred thousand candidates before any model runs.
Start with deterministic keys, because they are free and nearly always right. Normalize website domains (strip protocol, "www," and trailing paths) and treat an exact domain match as a strong candidate. Normalize company names by lowercasing, removing legal suffixes like "inc," "llc," "gmbh," and "ltd," and collapsing punctuation. For contacts, a lowercased, trimmed email is the key, with care for Gmail dots and plus addressing.
Exact matches on a normalized domain plus a matching country can merge automatically with a log entry. That alone typically clears the largest share of your duplicates, and you have not spent a single API call yet.
Stage Two: Let Embeddings Find What Strings Miss
Rules cannot know that "Alphabet" and "Google" are related, or that "St. Luke's Health System" and "Saint Lukes Hospital" are probably the same buyer. Embeddings can get close.
Build one text string per record that captures identity, not activity: normalized name, domain, city, country, and industry. Leave out notes, lifecycle stage, and owner, since those describe your relationship with the record rather than what the record is. Embed each string with a small embedding model and store the vectors in whatever you already run, whether that is pgvector, your warehouse's vector functions, or a dedicated store.
For each record, pull its top five nearest neighbors within the same country block. Pairs above a high cosine similarity go to review. Pairs below a low floor get dropped. Everything in between is the ambiguous middle, and that is the only place the expensive model belongs.
Two warnings. First, calibrate thresholds on your own data by labeling 200 pairs by hand, because similarity scores do not transfer between embedding models. Second, embeddings are bad at telling apart siblings: "Acme Logistics" and "Acme Labs" will look close. That is exactly why stage three exists.
Stage Three: The LLM as Adjudicator, Not Auto-Merger
Send each ambiguous pair to a model with both full records and a narrow instruction: decide whether these represent the same legal or buying entity. Require a structured response with a verdict of same, different, parent_subsidiary, or unsure, a confidence score, and a one-sentence reason that cites the fields it relied on.
The parent_subsidiary option matters more than people expect. Many "duplicates" are really a parent company and a division that buy separately. Merging them breaks territory assignment and inflates account size. Your CRM probably has a parent account field; use it instead of collapsing the records.
Route by verdict. same with high confidence goes to a merge queue that applies survivorship rules. different gets written to a do-not-match table so the pair never comes back. parent_subsidiary updates the hierarchy field. unsure and anything low-confidence lands in a human review queue with the model's reason already attached, which turns a five-minute research task into a ten-second decision.
Duplicate Detection Approaches Compared
| Approach | Catches Name Variants | Avoids False Merges | Cost per 10k Pairs | Explains Decisions |
|---|---|---|---|---|
| Exact match rules only | No | Yes | Near zero | Yes, by rule |
| Fuzzy string thresholds | Partly | Weak on similar names | Near zero | Score only |
| Embeddings only | Yes | Weak on siblings | Low | No |
| Rules plus embeddings plus LLM adjudication | Yes | Strong | Low to moderate | Yes, per pair |
Run the adjudication step through a batch endpoint. Nobody is waiting on dedupe results in real time, and the batch discount makes a full backlog pass cheap enough to repeat monthly.
Merging Without Regret
The model decides whether two records are the same. It should never decide which values survive. Survivorship is a business rule, and it belongs in code you can read.
Write it down explicitly: the oldest record ID survives so external references keep working, the most recently verified email wins, the owner on the record with the open opportunity wins, and activity history is unioned, not overwritten. Store every merge as a row with both original records serialized, the verdict, the model version, the prompt version, and a timestamp. If a merge turns out wrong, you can reverse it from that row instead of begging your CRM vendor's support team.
Then prevent the next wave. Run the same pipeline at the point of entry, so a new form fill or list import is checked against existing records before it becomes another duplicate. Prevention is far cheaper than cleanup.
Entity
- Normalize domains, company names, and emails, and auto-merge only exact domain plus country matches
- Build identity-only embedding strings and block nearest neighbor search by country
- Label 200 pairs by hand to set your upper and lower similarity thresholds
- Require a structured verdict with same, different, parent_subsidiary, and unsure options
- Write rejected pairs to a do-not-match table so they never resurface
- Encode survivorship rules in code, not in the prompt
- Log every merge with both original records so it can be reversed
- Run the pipeline at record creation, not just as a quarterly cleanup
Start With One Object
Do not try to resolve accounts, contacts, and leads in the same sprint. Start with accounts, because account duplicates do the most damage to routing, ABM, and reporting. Run stages one and two on a full export this week, measure how many candidate pairs land in the ambiguous middle, and adjudicate a sample of 500 before you touch production. Once the model's verdicts agree with your hand labels on that sample, turn on the merge queue with a human approving every batch for the first month.
Clean CRM data has always been the foundation every marketing system quietly depends on. The difference now is that the hard part, deciding whether two messy records describe the same company, finally has a tool that can do it with a reason attached.
Tags
LETSGROW Dev Team
Marketing Technology Experts
Ready to Apply This Insight?
Schedule a strategy call to map these ideas to your architecture, data, and operating model.
Schedule Strategy Call