Understanding the Unicode code point inspector — concept and principle
Every character is stored as a number called a Unicode code point, written U+ followed by at least four hex digits. An encoding then turns that number into bytes, usually UTF-8 (files, web, APIs) or UTF-16 (JavaScript, Java, Windows). This inspector shows, per character, the code point, the UTF-8 bytes and the UTF-16 units, which helps when debugging garbled text or checking what a database will store.
How it is calculated
- Code point: the number assigned by the Unicode Standard, U+0000 to U+10FFFF.
- UTF-8: 1 byte up to U+007F, 2 bytes up to U+07FF, 3 bytes up to U+FFFF, 4 bytes above. The first byte signals the length (0xxxxxxx, 110xxxxx, 1110xxxx, 11110xxx); each following byte is 10xxxxxx and carries 6 bits.
- UTF-16: one 16-bit unit inside the Basic Multilingual Plane (BMP, up to U+FFFF). Above it, subtract 0x10000 and split the 20 bits into a high surrogate 0xD800 + top 10 bits and a low surrogate 0xDC00 + bottom 10 bits.
- Counts: code points / UTF-16 units (JavaScript .length) / UTF-8 bytes.
The principle
Unicode separates identity from storage: the code point says which character it is, the encoding decides how many bytes to spend. UTF-8 keeps ASCII as one byte, and because continuation bytes always begin with 10, a reader can resynchronise anywhere in a stream. UTF-16 reaches the full 1,114,112 code points by using the reserved range U+D800–U+DFFF as surrogate pairs.
Worked example
- Input: 한😀A, three characters.
- 한 = U+D55C. Its bits 1101 010101 011100 fill the 3-byte pattern as ED 95 9C; in UTF-16 it is one unit, D55C.
- 😀 = U+1F600, outside the BMP. UTF-8: F0 9F 98 80. UTF-16: 0x1F600 − 0x10000 = 0xF600, top 10 bits 0x3D, bottom 10 bits 0x200, giving D83D DE00.
- A = U+0041: UTF-8 41, UTF-16 0041.
- Totals: 3 code points, 1 + 2 + 1 = 4 UTF-16 units, 3 + 4 + 1 = 8 bytes, shown as 3 / 4 / 8.
How to read the result
A UTF-16 length larger than the character count means the text contains characters beyond the BMP, usually emoji. Korean syllables (U+AC00–U+D7A3) always take 3 bytes in UTF-8 and one UTF-16 unit.
Common mistakes
- Assuming one character is one byte, or that JavaScript .length counts characters.
- Cutting a string inside a surrogate pair or a multi-byte UTF-8 sequence.
- Treating a skin-tone or family emoji as one code point; these are sequences joined by modifiers or U+200D.
Frequently asked questions
Why does MySQL reject emoji in a utf8 column?
Legacy utf8 (utf8mb3) stores at most 3 bytes per character, so 4-byte characters do not fit. Use utf8mb4.
Is UTF-16 always 2 bytes per character?
No. Characters above U+FFFF take 4 bytes as a surrogate pair.
Which encoding should I use for new files?
UTF-8 is the default for the web and most tools; it is compact for ASCII and has no byte-order issue.
Sources: The Unicode Consortium, The Unicode Standard, Chapters 2 and 3 · IETF RFC 3629, UTF-8, a transformation format of ISO 10646 · IETF RFC 2781, UTF-16, an encoding of ISO 10646