1.3.2 Unicode Coding Scheme

Introduction

Even extended ASCII (ASCII-8) does not include enough code combinations to support all written languages. Asian languages, for instance, require thousands of characters. This limitation gave rise to a new encoding standard known as Unicode (Universal coding system) that can support all the written languages.

Definition

Unicode is an international character-encoding system designed to facilitate the electronic exchange, processing, and display of written texts from diverse modern and classical languages.

It was published by the Unicode Consortium (a group of multilingual software manufacturers). The first version of Unicode was introduced in 1991.

Main Idea

Unicode assigns a unique number to each character that remains consistent across all systems that support Unicode. This means the same number represents the same character on any computer, in any country, regardless of the platform or software being used.

Core Concept

The Unicode Standard encompasses:

• Letters
• Digits
• Accents
• Punctuation marks
• Technical symbols for the world's major written languages
• Emoji and other symbols

All of these use a uniform encoding scheme. The latest version of Unicode includes over 100,000 characters.

Detailed Explanation

Unicode Encodings

Unicode defines multiple encodings of its single character set:

• UTF-7
• UTF-8
• UTF-16
• UTF-32

UTF stands for Unicode Transformation Format.

Conversion of data among these encodings is lossless, meaning it does not lose the quality or information.

Development of Unicode

• Unicode was originally a 2-byte (16-bit) character set.
• Unicode version 3, however, is a 4-byte (32-bit) code and is fully compatible with ASCII.
• All these encodings support encoding the same set of characters.

Table 4 (Annexure 4) shows a part of the Unicode character set after ASCII-8.

Intermediate Level

At the intermediate level, you should understand the difference between ASCII and Unicode.

Key point: The first 128 Unicode characters are the same as ASCII. This means ASCII text is automatically valid Unicode text.

Important Points

• - Unicode is a universal character encoding system.
• - It supports all written languages, including Asian languages.
• - Published by the Unicode Consortium.
• - First version: 1991.
• - Latest version: over 100,000 characters.
• - Encodings: UTF-7, UTF-8, UTF-16, UTF-32.
• - UTF-8 is the most popular on the internet.
• - Unicode is fully compatible with ASCII.
• - Conversion between encodings is lossless.

Remember

• - Unicode = Universal coding system.
• - First version: 1991.
• - Over 100,000 characters.
• - UTF = Unicode Transformation Format.
• - UTF-8 is most common on the internet.
• - First 128 Unicode characters = ASCII.
• - Conversion between encodings is lossless.

Common Mistakes

• - Thinking that Unicode is the same as ASCII (Unicode is much larger).
• - Confusing UTF-8, UTF-16, and UTF-32 (they are different encodings of the same character set).
• - Forgetting that Unicode is backward compatible with ASCII.
• - Thinking that Unicode only supports modern languages (it also supports classical languages).
• - Assuming that all Unicode characters take the same number of bytes (UTF-8 is variable-length).
• - Confusing the Unicode standard with its encodings (UTF-8 is an encoding, not the standard itself).

Quick Revision

• Unicode is an international character-encoding system.
• Supports all written languages.
• Published by the Unicode Consortium.
• First version: 1991.
• Over 100,000 characters.
• Encodings: UTF-7, UTF-8, UTF-16, UTF-32.
• UTF-8 is the most popular on the internet.
• Fully compatible with ASCII.
• Conversion between encodings is lossless.

60-Second Revision

• Unicode = universal character encoding.
• Supports all world languages.
• First version: 1991; over 100,000 characters.
• Encodings: UTF-7, UTF-8, UTF-16, UTF-32.
• UTF-8 = most popular on the internet.
• First 128 characters = ASCII.
• Warning: Unicode is not the same as ASCII; ASCII is a small subset of Unicode.