Two layers: the catalogue and the encoding
Unicode is a catalogue. It gives every character a code point, written U+ followed by a hexadecimal number: U+0041 is A, U+00E9 is é, U+20AC is the euro sign, U+1F600 is a grinning face. The space runs to U+10FFFF, which is room for over a million code points.
UTF-8 is an encoding: a rule for turning code points into bytes. The Unicode Consortium's FAQ describes it as a sequence of one to four 8-bit bytes per code point, the most widely used ASCII-compatible encoding form, and the most common on the web. UTF-16 uses one or two 16-bit units, UTF-32 always one 32-bit unit. Same catalogue, different byte layouts; a file "in Unicode" is really a file in one of these.
How UTF-8 lays out the bytes
RFC 3629 defines the pattern. The first byte says how long the sequence is; continuation bytes all start with the bits 10, so a reader dropped into the middle of a stream can find the next character boundary.
| Code points | Bytes | Bit pattern | What lives there |
|---|---|---|---|
| U+0000 to U+007F | 1 | 0xxxxxxx | ASCII: Latin letters, digits, punctuation |
| U+0080 to U+07FF | 2 | 110xxxxx 10xxxxxx | Latin accents, Greek, Cyrillic, Hebrew, Arabic |
| U+0800 to U+FFFF | 3 | 1110xxxx 10xxxxxx 10xxxxxx | Most of Chinese, Japanese and Korean, the euro sign, many symbols |
| U+10000 to U+10FFFF | 4 | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | Emoji, historic scripts, rare CJK characters |
The first row is the trick that made UTF-8 win: the RFC says it preserves the full US-ASCII range, so every ASCII character is the same single byte it always was, and any plain ASCII text is already valid UTF-8. A few characters encoded here:
| Character | Code point | UTF-8 bytes | Hex |
|---|---|---|---|
| A | U+0041 | 1 | 41 |
| é | U+00E9 | 2 | C3 A9 |
| € | U+20AC | 3 | E2 82 AC |
| 中 | U+4E2D | 3 | E4 B8 AD |
| 😀 | U+1F600 | 4 | F0 9F 98 80 |
The text to binary converter shows the bits of any text you paste, and the HTML entity converter turns characters into the € form that survives any encoding.
Why é turns into é
The bytes are only meaningful with the right rule. Save "é" as UTF-8 and you get C3 A9. Open the file with software that assumes a one-byte encoding such as Windows-1252, and each byte is shown as its own character: C3 is à and A9 is ©, giving é. The text is intact; the label is wrong. The fixes are all about the label: declare the encoding in the file or protocol (a meta charset of UTF-8 in HTML, a UTF-8 header in email and HTTP), save with the editor's UTF-8 option, and convert old files once rather than patching characters. The reverse mistake, reading Windows-1252 bytes as UTF-8, shows replacement characters (black diamonds with question marks) because the byte sequences are not valid UTF-8.
Why character counts disagree
Take the text "Zürich 🇨🇭". It is 9 characters as people count them, 11 UTF-16 code units (which is what most programming languages report as the string length), and 16 bytes in UTF-8. The flag is two code points that render as one symbol, each needing four bytes. So three honest limits give three answers:
- Character limits (a tweet, a form field) usually count code points or what the platform decides is a character; the character counter counts the way people read.
- Byte limits (a database column, an SMS, a URL) count the UTF-8 bytes; text with accents or emoji uses them up two to four times faster than English.
- Code-unit limits show up in JavaScript and Java string lengths, where an emoji counts as two.
Base64, which the Base64 encoder produces, adds a fourth number: it encodes the bytes, not the characters, so the same text is about a third longer again.
The byte order mark
U+FEFF at the very start of a file is the byte order mark. For UTF-16 and UTF-32 it tells the reader which byte comes first; for UTF-8, the Unicode FAQ says it is used only as a signature, and RFC 3629 gives its bytes as EF BB BF. Three invisible bytes before the first real character are harmless in a word processor and a problem in a shell script, a CSV header or a JSON file that a program parses strictly. Save UTF-8 without a BOM unless a specific Windows tool asks for one.
Rules of thumb
- Use UTF-8 everywhere: files, databases, HTML, email, APIs. Declare it explicitly rather than relying on defaults.
- Never fix mojibake by retyping characters; fix the declared encoding once and re-read the bytes.
- When a limit matters, ask which unit it counts: characters, bytes or code units.
- Save without a BOM unless a tool demands it, and treat EF BB BF at the start of a file as the explanation when a parser complains about the first line.
Frequently asked questions
What is the difference between Unicode and UTF-8?
Unicode is the catalogue: it assigns every character a number, the code point, from U+0000 to U+10FFFF. UTF-8 is one way to write those numbers as bytes, using one to four bytes per character. UTF-16 and UTF-32 are the other common encodings of the same catalogue. When a file "is Unicode", the real question is which encoding it uses.
Why is UTF-8 the default everywhere?
The Unicode Consortium's own FAQ calls it the most widely used ASCII-compatible encoding and the most common on the web. Every ASCII byte keeps its value, so old English-only software and formats keep working, and a plain ASCII file is already valid UTF-8. Everything else costs two to four bytes per character.
Why do I see é or ü instead of é or ü?
The text was saved as UTF-8 and read as a one-byte encoding such as Windows-1252. The two bytes of é (C3 A9) are shown as two separate characters. The fix is to tell the reader the right encoding, for a web page with a meta charset of UTF-8, rather than to change the text.
What is a BOM and should I use it?
The byte order mark is the character U+FEFF at the start of a file. RFC 3629 says it appears in UTF-8 as the bytes EF BB BF, and the Unicode FAQ says that in UTF-8 it serves only as a signature, not as a byte order indicator. Avoid it in UTF-8 files that a protocol or a script will parse, since it is three invisible bytes before the first real character.
Is one character always one byte?
Only in ASCII. In UTF-8 a character is one to four bytes, and some visible characters are several code points, such as a flag emoji made of two regional indicator symbols. That is why a character limit, a byte limit and a code-unit limit are three different numbers for the same text.
Sources
- Unicode Consortium FAQ: UTF-8, UTF-16, UTF-32 and BOM (one to four bytes per code point in UTF-8, the most widely used ASCII-compatible form and the most common on the web; the BOM as a signature)
- RFC 3629: UTF-8, a transformation format of ISO 10646 (the byte layout table, preservation of the US-ASCII range, the BOM bytes EF BB BF)