Character Encoding — Computer Organization & Architecture

Introduction

Computers only understand numbers, but humans communicate with text. Character encoding is the bridge between these two worlds — it assigns a unique number to every character (letter, digit, symbol) so that text can be stored, transmitted, and displayed by digital systems. From the simple ASCII code that assigns 65 to the letter 'A', to the Unicode standard that covers every writing system on Earth, character encoding is fundamental to how computers handle text.

Why Character Encoding Matters

When you type "Hello" on your keyboard, the computer doesn't store the letters — it stores numbers. Each character is mapped to a specific binary code. The encoding scheme determines which number represents which character. If two systems use different encodings, text appears garbled — those mysterious "????" or "é" characters you sometimes see are encoding mismatches.

ASCII (American Standard Code for Information Interchange)

Overview

Developed in 1963, ASCII is the foundation of text encoding. It uses 7 bits to represent 128 characters (0-127).

Character Categories

• 0-31: Control characters (non-printable) — newline, tab, bell, etc.
• 32-47: Punctuation and symbols — space, !, @, #
• 48-57: Digits 0-9
• 65-90: Uppercase A-Z
• 97-122: Lowercase a-z
• 123-127: More punctuation — {, }, |, ~

Key ASCII Values to Remember

Clever Design Features

• Digits 0-9 have codes 48-57 → the binary representations end in the BCD value of the digit
• Uppercase and lowercase differ by exactly 32 (one bit flip: bit 5)
• 'A'=65, 'a'=97, difference = 32 = 2⁵. So toggling bit 5 switches case!

Extended ASCII (8-bit)

Since most systems use 8-bit bytes, various "extended ASCII" schemes use codes 128-255 for additional characters (accented letters, box-drawing characters). But different systems used this range differently — creating compatibility chaos.

EBCDIC

Extended Binary Coded Decimal Interchange Code was IBM's encoding for mainframes. It uses 8 bits but with a completely different mapping than ASCII. Letters aren't contiguous (there are gaps), making string processing more complex. EBCDIC is still used on IBM mainframes but is rare elsewhere.

Unicode: The Universal Standard

The Problem Unicode Solves

ASCII only covers English. What about Chinese (50,000+ characters), Arabic, Hindi, Japanese, Korean, emoji? The world needed one encoding that covers ALL writing systems.

Unicode Design

Unicode assigns a unique code point to every character in every writing system. Code points are written as U+XXXX (hexadecimal).

• U+0041 = A (Latin capital A)
• U+03B1 = α (Greek alpha)
• U+4E2D = 中 (Chinese "middle")
• U+1F600 = 😀 (Grinning face emoji)

Unicode currently defines over 149,000 characters covering 161 scripts.

Unicode Encoding Forms

Unicode defines code points, but how are they stored in memory? That's where encoding forms come in:

UTF-8 (The Web Standard)

UTF-8 is the dominant encoding on the internet (98%+ of web pages). It's a variable-length encoding — different characters use different numbers of bytes:

Why UTF-8 is Brilliant

1. Backward compatible with ASCII: ASCII text is valid UTF-8 (no conversion needed)
2. Self-synchronizing: You can find character boundaries by looking at byte patterns
3. No byte-order issues: Reads the same on big-endian and little-endian systems
4. Compact for English: Uses 1 byte for ASCII characters
5. Complete: Can represent all Unicode characters

UTF-8 Example: Encoding "Héllo"

UTF-16

Uses 2 bytes for common characters (Basic Multilingual Plane) and 4 bytes for rare characters (supplementary planes via surrogate pairs). Used internally by Windows and Java.

UTF-32

Fixed-width 4 bytes per character. Simple indexing but wasteful of space. Used internally by some Unix systems for processing convenience.

Comparison of Encodings

Practical Impact on Computer Organization

Memory Considerations

Character encoding affects memory usage. A document in UTF-8 might be 1MB, but the same content in UTF-32 could be 2-4MB. Memory-constrained systems (embedded devices) often use ASCII-only to save space.

Processor Operations

String comparison, sorting, and searching depend on the encoding. Fixed-width encodings (ASCII, UTF-32) allow direct indexing (character n is at byte offset n×width), while variable-width encodings (UTF-8, UTF-16) require scanning from the beginning.

I/O and Communication

When data moves between systems, encoding must be agreed upon. HTTP headers specify encoding (Content-Type: text/html; charset=utf-8), file systems store encoding metadata, and network protocols define character representation.

Key Takeaways

1. Character encoding maps human text characters to binary numbers
2. ASCII (7-bit, 128 characters) is the foundation but covers only English
3. Unicode assigns a unique code point to every character in every writing system
4. UTF-8 is the dominant encoding — variable-length, ASCII-compatible, and efficient
5. Encoding mismatches cause garbled text — always be explicit about encoding
6. The choice of encoding affects memory usage, processing speed, and compatibility
7. Modern systems overwhelmingly use UTF-8 for storage and interchange