Unicode and UTF-8 Explained: Fixing Encoding Problems

The short answer: where garbled text comes from

Computers store bytes, not letters. An encoding is the rule that turns bytes into characters. If text is saved in one encoding and opened in another, instead of the Russian word "Привет" you get "Привет". This is called mojibake.

The fix is almost always the same: use UTF-8 everywhere and state the encoding explicitly when reading and writing.

Unicode, code points and UTF-8

• Unicode is a large table that gives every character a number, a code point. Cyrillic "А" is U+0410, Latin a is U+0061, the Uzbek mark ‘ in o‘ is U+2018.
• UTF-8 is a way to write a code point as bytes. Latin letters take 1 byte, Cyrillic 2, many punctuation marks and symbols 3, emoji 4.

Windows-1251 is an old single-byte encoding for Cyrillic. In it "А" is one byte, C0. It still shows up in exports from legacy systems, Excel CSV files and old databases.

How the corruption happens

Take "П": in UTF-8 it is the bytes D0 9F. A program that thinks the file is Windows-1251 sees two separate characters: D0 is "Р" and 9F is "џ". That is the "Рџ" at the start of the broken word.

The reverse case: a Windows-1251 file opened as UTF-8. Those bytes do not form valid sequences, and you get � replacement characters instead of text.

How to fix it in practice

Read and write files with an explicit encoding:

with open("data.csv", encoding="utf-8") as f:
    text = f.read()

Convert legacy files:

iconv -f WINDOWS-1251 -t UTF-8 old.csv > new.csv

Repair an already broken string if you know how it broke:

broken = "Привет"
fixed = broken.encode("cp1251").decode("utf-8")  # "Привет"

Check every layer:

• HTML: <meta charset="utf-8"> and the header Content-Type: text/html; charset=utf-8.
• MySQL: the utf8mb4 charset, not utf8, which stores at most 3 bytes and rejects emoji.
• PostgreSQL: a database with UTF8 encoding.
• CSV for Excel: save as UTF-8 with BOM (encoding="utf-8-sig" in Python), otherwise Excel may open Cyrillic incorrectly.
• Code editor and Git: files in UTF-8.

Uzbek o‘ and g‘: a separate trap

The apostrophe in o‘, g‘ and in words like ma’lumot can be typed with different characters: ' (U+0027), ` (U+0060), ‘ (U+2018), ’ (U+2019), ʻ (U+02BB). They look alike, but to a computer they are different characters. The consequences:

• searching for "o‘zbek" does not find "o'zbek";
• duplicates in directories, customer lists and addresses;
• different URLs and slugs for the same word.

What to do: pick one variant for the project and normalize input on save, replacing the other variants with the chosen one. For search, store an extra normalized field.

One more detail is string length. "o‘" is 2 code points but 4 bytes in UTF-8. If a database column is limited in bytes rather than characters, text can be cut in the middle of a character.

Unicode normalization

Some letters can be written two ways: "й" as one code point, or "и" plus a combining mark. They look identical, yet string comparison returns False. Bring text to one form:

import unicodedata
clean = unicodedata.normalize("NFC", text)