When Invisible Zero-Width Spaces Break Database Imports
Two SKU strings can look identical on screen yet fail equality checks in PostgreSQL or JavaScript:
"SKU-8921" !== "SKU-\u200B8921"
When product catalogs or legal paragraphs are copied out of PDFs, Word docs, or chat apps, invisible Unicode characters like the Zero-Width Space (U+200B) and byte-order marks (U+FEFF) ride along silently — breaking unique indexes, search hashes, and CSV parsers.
The Hidden Gremlins of Copy-Pasted Text
When text travels between PDFs, Microsoft Word documents, web browsers, and chat clients, it picks up non-printable formatting artifacts:
- Smart / Curly Quotes (
“”vs""): Word processors automatically replace standard ASCII quotes (0x22) with typographic quotes, breaking SQL queries and JSON parsers. - Non-Breaking Spaces (
/ U+00A0): Looks like a space, but fails regex tests like/\s/depending on encoding flags. - Mixed Line Endings (CRLF vs LF): Windows (
\r\n) vs Unix (\n) line breaks cause diff mismatches and parser desynchronization.
Clean & Sanitize Dirty Text Online
Remove duplicate spaces, strip non-printable Unicode characters, standardize quotes, and normalize line breaks in one click.
The 4-Step Text Sanitization Pipeline
- Strip Zero-Width & Control Characters: Purge
[\u200B-\u200D\uFEFF]. - ASCII Normalization: Convert fancy typographic quotes, dashes (em-dash —, en-dash –), and ellipses into standard ASCII equivalents.
- Collapse Whitespace: Replace multiple continuous spaces or tabs with a single clean space.
- Trim Boundaries: Strip leading and trailing whitespace from every line.
