Normalizing free-text fields is where the real accuracy hides
Humans have been typing into your fields for years, each in their own way. Before a model can reason over that text, someone has to turn a thousand spellings of the same thing into one.
Normalizing free-text fields is where the real accuracy hides
Give a hundred people an empty text box and ask for a US state. You will get California, CA, Calif., california, and a few with a trailing space. To a person these are one answer. To a model, and to any join or count underneath it, they are five different things.
The field that means one thing and stores five
An engineering lead who has moved close to three billion records between business systems described this better than any style guide. Pull a customer record out of one system and the shipping state reads “California.” The system she is loading it into wants “CA.” That is one field, in one record. Now multiply it by every free-text field, every system, and every person who typed into those fields their own way for a decade. The “clean” data you imagined does not exist. What exists is years of human variation stored as if it were structured.
Free text is where humans put the things a form did not constrain. State, city, company name, job title, product description, notes. Because nothing forced a standard, no standard emerged. The data is technically full and functionally inconsistent, and a model asked to reason over it treats every variation as distinct. Group customers by state and California and CA land in separate buckets. The count is wrong, and nothing errored to tell you.
The catalog example is the same story, worse
Her product catalog case shows how deep this goes. A demo shows a model reading “Nike red shirt medium” and neatly splitting it into brand, color, and size. Then production hands it fifty thousand SKUs from an apparel store where the variants were entered inconsistently across a decade. “Red” is also “crimson” and “scarlet” and “RD.” Size is “medium,” “M,” “Med,” and “Adult M.” The clean parser meets a field that fifty different employees filled in fifty different ways, and its accuracy falls off a cliff the demo never approached.
This is the work that decides whether a model is reliable, and it happens before the model runs. AI features generally need on the order of 85% accuracy to hold up. You cannot reach that on text where the same value hides behind a dozen spellings, because every unnormalized variant is a small wrong grouping, and they add up fast.
Normalization is a mapping someone has to build
Turning “California” into “CA” across every record sounds trivial until you try it at scale. Someone has to build the mapping. Someone has to decide that “Calif.” and “california “ and a misspelled “Californa” all resolve to the same value, and someone has to handle the entries that are genuinely ambiguous. For the SKU catalog, someone has to define the vocabulary of colors and sizes and map thousands of freeform strings onto it. This is deliberate, iterative work, and it does not fit in a demo timeline.
Which is exactly why pilots skip it. The pilot runs on a slice where a human already normalized the fields, the model looks sharp, and the normalization debt stays invisible until production pours in its real, unstandardized text. The migration lead builds this into her process on purpose. She runs the core data first, has a human reconcile the mappings, and only then reruns the tool for the incremental records, because the mapping is never right on the first pass and someone accountable has to tune it.
The accuracy was in the boring step all along
Teams chase model accuracy in the model, tuning prompts and swapping versions, when a large share of the accuracy was sitting in the free-text fields the whole time. Normalize the state field and the state grouping is suddenly right. Normalize the catalog and the parser works. The gains that felt like they should come from a smarter model often come from cleaner input, and cleaner input is unglamorous, upfront, human-guided work.
How we approach it at Density Labs
In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we look at the free-text fields a feature depends on and ask whether they are normalized or just full. Where the same value can be written a dozen ways, state, product attributes, company names, we flag it, because that variation becomes wrong groupings and wrong counts the model will quietly inherit. Building the mapping in week one is cheap. Discovering its absence in a stalled pilot is not.
The accuracy you are hunting for in the model is often waiting in a text field nobody standardized.