Invisible characters, and why your lookup fails

A key that is obviously right does not match. A join returns nothing. A translation string is reported missing while you are looking straight at it. You copy both values into a text editor and they are still identical.

They are not identical. One of them carries a character that occupies no space and draws nothing, and string comparison does not care that you cannot see it.

Which characters count as invisible?

There is a specific list, and it is shorter than it feels. Grouped by what they were designed for:

U+00A0 non-breaking space — a space that browsers and word processors will not break a line at. It looks exactly like a normal space and is a different character.
U+00AD soft hyphen — a hint about where a word may be hyphenated. Invisible unless the line happens to break there.
U+200B zero-width space, U+2060 word joiner — line-breaking hints with no width at all.
U+200C zero-width non-joiner, U+200D zero-width joiner — control how adjacent letters connect. Legitimate in Arabic and Persian, and U+200D is also what glues modern emoji sequences together.
U+200E / U+200F direction marks and U+202A–U+202E embedding and override controls — force text direction for mixed left-to-right and right-to-left content.
U+FEFF — the byte order mark when it is at the start of a file, and a zero-width no-break space anywhere else.
U+3000 ideographic space — the full-width space of Chinese, Japanese and Korean typesetting. This one you can see; it is simply not the space you think it is.

Where each one comes from

The provenance is worth knowing, because it tells you how many more you have.

Pasting from a web page or a word processor brings non-breaking spaces. HTML authors use them to hold things apart, and the paste keeps them. If you got one this way, you have as many as the source had.

Pasting from a PDF brings soft hyphens and zero-width spaces, because the layout engine put them there to justify the text and the extraction handed them over with everything else.

Right-to-left content brings direction marks, often added by whoever typed it so that mixed Arabic and Latin text displayed correctly in their editor. They were solving a real problem; the marks just should not have survived into a key.

A file concatenated from several exports brings U+FEFF in the middle of it — each part had its own byte order mark, and only the first one was at the start.

CJK text typed with an input method brings ideographic spaces, because that is the space key.

Why nothing looks wrong

String comparison is character by character. "item_mug" and "item_mug" differ in length, so they are not equal, and there is nothing more to it than that. No lookup, join, dictionary or === will treat them as the same.

What makes it feel like a mystery is that every layer you use to investigate hides the evidence. The terminal renders nothing. The browser renders nothing. A diff shows two lines that look the same. Copying the value out carries the character with it, so pasting it somewhere else reproduces the problem rather than exposing it.

This is also why “just trim it” is not enough. trim() removes whitespace, and most of these characters are not whitespace by that definition — a zero-width space survives a trim intact.

One related failure that is not on the list above: two values can differ by Unicode normalisation rather than by an invisible character. é can be one code point or an e followed by a combining accent, and those are different strings that render identically. Nothing in the invisible-character family will find that one, because both forms are visible; it needs a normalisation pass instead.

The one that is usually not a mistake

The ideographic space, U+3000, deserves separate treatment, and the editor on this site classifies it differently from the rest for that reason: it is reported as information rather than as a warning.

In Chinese, Japanese and Korean text a full-width space is often exactly what the author meant. It is the correct space for that script, it comes from the standard input method, and stripping it changes the text.

So it is worth knowing it is there — a full-width space in an identifier column is almost certainly wrong — while a bulk “remove all spaces like this” across a file of Japanese descriptions would quietly damage the content. The distinction is not whether the character is suspicious, but which column it turned up in.

Finding and removing them

The requirement is a tool that renders them, because by definition you cannot search for something you cannot see or type. Once you can see them, the sequence is:

Find every instance, not the one that broke. The value that failed is the one you noticed, not the only one affected. Whatever pasted a non-breaking space once pasted it fifty times.
Decide per character class, not per cell. Zero-width characters in a key column are always wrong. Non-breaking spaces usually want to become normal spaces rather than disappear. Ideographic spaces in descriptive text should stay.
Apply it across the file in one operation, so the result is consistent and reversible in one step.
Re-check afterwards. Removing a zero-width space can reveal a trailing normal space that was hiding behind it.

For the wider set of problems that break a file, how to fix a broken TSV puts this one in order among the others — encoding first, structure next, characters last.

Doing it here

The editor on this site draws whitespace in the grid where it sits, so a trailing space is something you see rather than something you deduce. Invisible characters are reported with their exact code point, counted per cell, and the count in the cell inspector comes from the same function as the count in the review panel — two definitions of “invisible” would eventually disagree, and this is the one feature where that must not happen.

Each class is a separate finding you can fix across the whole file in a single undo step, so trying one costs nothing. The ideographic space is reported as information rather than as an error, because in a column of Japanese text it is probably correct.

Nothing is uploaded: the file is read, checked and written by your browser.