Character Encoding — Computer Fundamentals

Introduction

When you type the letter "A" on your keyboard, the computer does not store an actual letter — it stores a number. When that number is later displayed on screen, the computer looks up which character that number represents and draws the appropriate shape. This mapping between numbers and characters is called character encoding, and it is one of those fundamental concepts that affects everything from emails to websites to databases — yet most people never think about it until something goes wrong.

You have probably seen garbled text before — strange symbols like é, ’, or □□□ appearing where normal text should be. This happens when text encoded in one system is decoded using a different system — like trying to read a message that was encoded with a different key. Understanding character encoding helps you prevent and fix these problems.

Why Computers Need Encoding

Computers process everything as numbers — specifically as binary numbers (sequences of 0s and 1s). Text, images, audio, video — all must be represented numerically. For text, we need a system that assigns a specific number to each character: A=65, B=66, a=97, b=98, 0=48, and so on.

But which characters need numbers? Just English letters and digits? What about French accents (é, ü), Chinese characters (中, 文), Arabic script (ع, م), mathematical symbols (∑, ∞), emojis (😀, 🎉), and thousands of other characters used worldwide? Different encoding systems answer this question differently — which is why multiple encoding standards exist.

ASCII — The Foundation

ASCII (American Standard Code for Information Interchange), developed in the 1960s, was one of the first widely adopted character encoding standards. It uses 7 bits to represent each character, giving 128 possible values (0-127).

ASCII defines 95 printable characters: uppercase letters A-Z (codes 65-90), lowercase letters a-z (codes 97-122), digits 0-9 (codes 48-57), punctuation and symbols (like @, #, $, +, =), and the space character (code 32). The remaining 33 codes (0-31 and 127) are control characters — non-printable codes for things like new line, tab, and backspace.

ASCII works perfectly for English text but has a critical limitation: it only supports 128 characters. This means no accented characters (French, German, Spanish), no Asian characters (Chinese, Japanese, Korean), no Arabic or Hebrew, and no special symbols beyond basic punctuation. As computers spread globally, this limitation became a serious problem.

Extended ASCII and Code Pages

To address ASCII's limitations, various "extended ASCII" schemes used 8 bits (allowing 256 characters) — adding 128 additional characters for specific languages. But different regions used different extensions: Western Europe used ISO-8859-1 (adding characters like é, ñ, ü), Eastern Europe used ISO-8859-2 (adding characters like ą, č, ž), and so on.

This created chaos — the same byte value meant different characters in different code pages. A document written in one region would display garbled text in another if the wrong code page was applied. And even with 256 characters, representing Asian languages (with thousands of characters) remained impossible.

Unicode — The Universal Solution

Unicode was developed starting in the late 1980s to solve this problem once and for all by providing a single, universal character set containing every character from every writing system in the world. As of 2024, Unicode defines over 149,000 characters covering 161 scripts, plus thousands of symbols, emojis, and special-purpose characters.

Unicode assigns each character a unique code point — a number in the format U+XXXX. For example: U+0041 is A, U+0042 is B, U+4E2D is 中 (Chinese character for "middle"), U+0905 is अ (Hindi), and U+1F600 is 😀 (grinning face emoji).

UTF-8 — Unicode in Practice

Unicode defines what number each character gets, but UTF-8 defines how those numbers are stored in computer memory. UTF-8 is a variable-width encoding — characters use 1 to 4 bytes depending on their code point value.

Standard ASCII characters (U+0000 to U+007F) use just 1 byte — making UTF-8 backwards-compatible with ASCII. An ASCII text file is already valid UTF-8. Common accented characters use 2 bytes. Most Asian characters use 3 bytes. Rare characters and emojis use 4 bytes.

UTF-8 is now the dominant encoding on the web (over 98% of websites use it) because it is efficient for English text (one byte per character, same as ASCII), supports all Unicode characters, is backwards-compatible with ASCII, and handles multilingual content in a single document.

Practical Implications

When creating text files, databases, or web pages, always use UTF-8 unless you have a specific reason not to. In HTML, declare your encoding in the head section. In databases, configure tables to use UTF-8. When reading files in programming, specify the encoding explicitly rather than relying on defaults.

Key Takeaways

• Character encoding maps characters to numbers so computers can store and process text
• ASCII uses 7 bits for 128 characters — sufficient for English but inadequate globally
• Unicode provides a universal standard covering all world writing systems (149,000+ characters)
• UTF-8 is the dominant Unicode encoding — variable-width, efficient, and backwards-compatible with ASCII
• Encoding mismatches cause garbled text — always be explicit about which encoding you use
• Always use UTF-8 for new projects unless you have specific requirements otherwise
• Understanding encoding prevents and diagnoses text display problems
• This knowledge is essential for web development, database design, and internationalization