From Wikipedia, the free encyclopedia

Unicode normalization is a form of text normalization that transforms equivalent characters or sequences of characters into a consistent underlying representation so that they may be easily compared. Normalization is important when comparing text strings for searching and sorting (collation).

Underlying Unicode's normalization methods is the concept of character composition and decomposition. Character composition is the process of combining simpler characters into fewer precomposed characters, such as the n character and the combining ~ character into the single ñ character. Decomposition is the opposite process, breaking precomposed characters back into their component pieces.

Unicode composes combining characters and decomposes compatibility characters based on what it calls equivalence. Unicode has two standards of varying breadth for this equivalence: canonical, which distinguishes between functionally and what should be visually indistinguishable characters, and other compatibility characters, which may be visually and even semantically distinct. See the articles on Unicode equivalence and Unicode compatibility characters for more information.

Unicode defines four normalization standards.

All the above methods will standardize the order in which decomposed characters appear, even sequences that were already decomposed prior to normalization. They may also replace characters or sequences with equivalent characters or sequences even if it doesn't result in the number of characters changing. These are done to achieve the consistency in encoding required for normalization.

• Unicode
• Unicode equivalence
• Unicode compatibility characters
• Precomposed character
• Ligature (typography)

• Unicode Standard Annex #15: Unicode Normalization Forms
• utf8proc: A library for normalization of Unicode texts