This article needs additional citations for verification. (April 2010) (Learn how and when to remove this template message)
|Standard||ISO 2022, ECMA 35, JIS X 0202|
|Transforms / Encodes||US-ASCII and, depending on implementation:|
|Succeeded by||ISO 10646 (Unicode)|
- a technique for including multiple character sets in a single character encoding system, and
- a technique for representing these character sets in both 7 and 8 bit systems using the same encoding.
Many of the character sets included as ISO/IEC 2022 encodings are 'double byte' encodings where two bytes correspond to a single character. This makes ISO-2022 a variable width encoding. But a specific implementation does not have to implement all of the standard; the conformance level and the supported character sets are defined by the implementation.
Many languages or language families not based on the Latin alphabet such as Greek, Cyrillic, Arabic, or Hebrew have historically been represented on computers with different 8-bit extended ASCII encodings. Written East Asian languages, specifically Chinese, Japanese, and Korean, use far more characters than can be represented in an 8-bit computer byte and were first represented on computers with language-specific double byte encodings.
ISO/IEC 2022 was developed as a technique to attack both of these problems: to represent characters in multiple character sets within a single character encoding, and to represent large character sets.
A second requirement of ISO-2022 was that it should be compatible with 7-bit communication channels. So even though ISO-2022 is an 8-bit character set any 8-bit sequence can be reencoded to use only 7-bits without loss and normally only a small increase in size.
To represent multiple character sets, the ISO/IEC 2022 character encodings include escape sequences which indicate the character set for characters which follow. The escape sequences are registered with ISO and follow the patterns defined within the standard. These character encodings require data to be processed sequentially in a forward direction since the correct interpretation of the data depends on previously encountered escape sequences. Note, however, that other standards such as ISO-2022-JP may impose extra conditions such as the current character set is reset to US-ASCII before the end of a line.
To represent large character sets, ISO/IEC 2022 builds on ISO/IEC 646's property that one seven bit character will normally define 94 graphic (printable) characters (in addition to space and 33 control characters). Using two bytes, it is thus possible to represent up to 8836 (94×94) characters; and, using three bytes, up to 830584 (94×94×94) characters. Though the standard defines it, no registered character set uses three bytes (although EUC-TW's unregistered G2 is). For the two-byte character sets, the code point of each character is normally specified in so-called kuten (Japanese: 区点) form (sometimes called qūwèi (Chinese: 区位), especially when dealing with GB2312 and related standards), which specifies a zone (区, Japanese: ku, Chinese: qū), and the point (Japanese: 点 ten) or position (Chinese: 位 wèi) of that character within the zone.
The escape sequences therefore do not only declare which character set is being used, but also, by knowing the properties of these character sets, know whether a 94-, 96-, 8836-, or 830584-character (or some other sized) encoding is being dealt with.
In practice, the escape sequences declaring the national character sets may be absent if context or convention dictates that a certain national character set is to be used. For example, ISO-8859-1 states that no defining escape sequence is needed and RFC 1922, which defines ISO-2022-CN, allows ISO-2022 SHIFT characters to be used without explicit use of escape sequences.
The ISO-2022 definitions of the ISO-8859-X character sets are specific fixed combinations of the components that form ISO-2022. Specifically the lower control characters (C0) the US-ASCII character set (in GL) and the upper control characters (C1) are standard and the high characters (GR) are defined for each of the ISO-8859-X variants; for example ISO-8859-1 is defined by the combination of ISO-IR-1, ISO-IR-6, ISO-IR-77 and ISO-IR-100 with no shifts or character changes allowed.
Although ISO/IEC 2022 character sets using control sequences are still in common use, particularly ISO-2022-JP, most modern e-mail applications are converting to use the simpler Unicode transforms such as UTF-8. The encodings that don't use control sequences, such as the ISO-8859 sets are still very common.
ISO/IEC 2022 coding specifies a two-layer mapping between character codes and displayed characters. Escape sequences allow any of a large registry of graphic character sets to be "designated" into one of four working sets, named G0 through G3, and shorter control sequences specify the working set that is "invoked" to interpret bytes in the stream.
Character codes from the 7-bit ASCII graphic range (0x20–0x7F), being on the left side of a character code table, are referred to as "GL" codes (with "GL" standing for "graphics left") while codes from the "high ASCII" range (0xA0–0xFF), if available, are referred to as the "GR" codes ("graphics right").
By default, GL codes specify G0 characters, and GR codes specify G1 characters, but this may be modified with control codes or by prior agreement:
Locking shift zero
|GL encodes G0 from now on|
Locking shift one
|GL encodes G1 from now on|
||LS2||Locking shift two||GL encodes G2 from now on|
||LS3||Locking shift three||GL encodes G3 from now on|
||SS2||Single shift two||GL encodes G2 for next character only|
||SS3||Single shift three||GL encodes G3 for next character only|
||LS1R||Locking shift one right||GR encodes G1 from now on|
||LS2R||Locking shift two right||GR encodes G2 from now on|
||LS3R||Locking shift three right||GR encodes G3 from now on|
Each of the four working sets may be a 94-character set or a 94n-character set. Additionally, G1 through G3 may be a 96- or 96n-character set. When one of the latter is invoked in the GL region, the space and delete characters (codes 0x20 and 0x7F) are not available.
There are additional (rarely used) features for switching control character sets, but this is a single-level lookup: the 0x00–0x1F range is the C0 control character set, the 0x80–0x9F range is the C1 control character set, and there are escape sequences which switch in various alternatives. It is required that any C0 character set include the ESC character at position 0x1B, so that further changes are possible.
As seen in the SS2 and SS3 examples above, single control characters from the C1 control character set may be invoked using only 7 bits using the sequences
ESC 0x40 (@) through
ESC 0x5F (_). Additional control functions are assigned in the range
ESC 0x60 (`) through
ESC 0x7E (~). While this article describes escape sequences using the corresponding ASCII characters, they are actually defined in terms of byte values, and the graphic assigned to that byte value may be altered without affecting the control sequence.
Escape sequences to designate character sets take the form
ESC I [I...] F, where there are one or more intermediate I bytes from the range 0x20–0x2F, and a final F byte from the range 0x40–0x7F. (The range 0x30–0x3F is reserved for private-use F bytes.) The I bytes identify the type of character set and the working set it is to be designated to, while the F byte identifies the character set itself.
||CZD||C0-designate||F selects a C0 control character set to be used.|
||C1D||C1-designate||F selects a C1 control character set to be used.|
||DOCS||Designate other coding system||F selects an 8-bit code; use |
||DOCS||Designate other coding system||F selects an 8-bit code; there is no standard way to return. E.g. |
||IRR||Identify revised registration||F, adjusted to the range 1-63, indicates which revision of the immediately-following registration is needed, so that old systems know that they are old.|
||GZD4||G0-designate 94-set||F selects a 94-character set to be used for G0.|
||G1D4||G1-designate 94-set||F selects a 94-character set to be used for G1.|
||G2D4||G2-designate 94-set||F selects a 94-character set to be used for G2.|
||G3D4||G3-designate 94-set||F selects a 94-character set to be used for G3.|
||G1D6||G1-designate 96-set||F selects a 96-character set to be used for G1.|
||G2D6||G2-designate 96-set||F selects a 96-character set to be used for G2.|
||G3D6||G3-designate 96-set||F selects a 96-character set to be used for G3.|
||GZDM4||G0-designate multibyte 94-set||F selects a 94n-character set to be used for G0.|
||G1DM4||G1-designate multibyte 94-set||F selects a 94n-character set to be used for G1.|
||G2DM4||G2-designate multibyte 94-set||F selects a 94n-character set to be used for G2.|
||G3DM4||G3-designate multibyte 94-set||F selects a 94n-character set to be used for G3.|
||G1DM6||G1-designate multibyte 96-set||F selects a 96n-character set to be used for G1.|
||G2DM6||G2-designate multibyte 96-set||F selects a 96n-character set to be used for G2.|
||G3DM6||G3-designate multibyte 96-set||F selects a 96n-character set to be used for G3.|
Note that the registry of F bytes is independent for the different types. The 94-character graphic set designated by
ESC ( A through
ESC + A is not related in any way to the 96-character set designated by
ESC - A through
ESC / A. And neither of those is related to the 94n-character set designated by
ESC $ ( A through
ESC $ + A, and so on; the final bytes must be interpreted in context. (Indeed, without any intermediate bytes,
ESC A is a way of specifying the C1 control code 0x81.)
Also note that C0 and C1 control character sets are independent; the C0 control character set designated by
ESC ! A (which happens to be the NATS control set for newspaper text transmission) is not the same as the C1 control character set designated by
ESC " A (the CCITT attribute control set for Videotex).
Additional I bytes may be added before the F byte to extend the F byte range. This is currently only used with 94-character sets, where codes of the form
ESC ( ! F have been assigned. At the other extreme, no multibyte 96-sets have been registered, so the sequences above are strictly theoretical.
ISO/IEC 2022 character sets
Character encodings using ISO/IEC 2022 mechanism include:
- ISO-2022-JP. A widely used encoding for Japanese. Starts in ASCII and includes the following escape sequences
- ISO-2022-JP-1. The same as ISO-2022-JP with one additional escape sequence
ESC $ ( Dto switch to JIS X 0212-1990 (2 bytes per character)
- ISO-2022-JP-2. A multilingual extension of ISO-2022-JP. The same as ISO-2022-JP-1 with the following additional escape sequences 
ESC $ Ato switch to GB 2312-1980 (2 bytes per character)
ESC $ ( Cto switch to KS X 1001-1992 (2 bytes per character)
ESC . Ato switch to ISO/IEC 8859-1 high part, Extended Latin 1 set (1 byte per character) [designated to G2]
ESC . Fto switch to ISO/IEC 8859-7 high part, Basic Greek set (1 byte per character) [designated to G2]
- ISO-2022-JP-3. The same as ISO-2022-JP with three additional escape sequences
- ISO-2022-JP-2004. The same as ISO-2022-JP-3 with one additional escape sequence
ESC $ ( Qto switch to JIS X 0213-2004 Plane 1 (2 bytes per character)
- ISO-2022-KR. An encoding for Korean.
- ISO-2022-CN. An encoding for Chinese.
- ISO-2022-CN-EXT. The same as ISO-2022-CN with six additional escape sequences
ESC $ ) Eto switch to ISO-IR-165 (2 bytes per character) [designated to G1]
ESC $ + Ito switch to CNS 11643-1992 Plane 3 (2 bytes per character) [designated to G3]
ESC $ + Jto switch to CNS 11643-1992 Plane 4 (2 bytes per character) [designated to G3]
ESC $ + Kto switch to CNS 11643-1992 Plane 5 (2 bytes per character) [designated to G3]
ESC $ + Lto switch to CNS 11643-1992 Plane 6 (2 bytes per character) [designated to G3]
ESC $ + Mto switch to CNS 11643-1992 Plane 7 (2 bytes per character) [designated to G3]
The character after the
ESC (for single-byte character sets) or
ESC $ (for multi-byte character sets) specifies the type of character set and working set that is designated to. In the above examples, the character
( (0x28) designates a 94-character set to the G0 character set. This may be replaced by
+ (0x29–0x2B) to designate to the G1–G3 character sets.
Two of the codes above are 96-character codes, and in the above examples, the character
- (0x2D) designates to the G1 character set. This may be replaced with
/ (0x2E or 0x2F) to designate to the G2 or G3 character sets. As mentioned earlier, a 96-character set may not be designated to the G0 set.
There are three special cases for multi-byte codes. The code sequences
ESC $ @,
ESC $ A, and
ESC $ B were all registered before the ISO/IEC 2022 standard was finalized, so must be accepted as synonyms for the sequences
ESC $ ( @ through
ESC $ ( B to designate to the G0 character set. The latter form may also be used, and may be adapted by changing the
( character to designate to the G1 through G3 character sets.
The standard also defines a way to specify coding systems that do not follow its own structure. Of particular interest, the sequence
ESC % G designates the UTF-8 coding system, which does not reserve the range 0x80–0x9F for control characters.
Comparison with other encodings
- As ISO/IEC 2022's entire range of 94-set graphical character encodings can be delegated to GL, the available glyphs are not significantly limited by an inability to represent GR and C1, such as in a system limited to 7-bit encodings. It accordingly enables the representation of large set of characters in such a system. Generally, this 7-bit compatibility is not really an advantage, except for backwards compatibility with older systems. The vast majority of modern computers use 8 bits for each byte.
- As compared to Unicode, ISO/IEC 2022 sidesteps Han unification by using sequence codes to switch between discrete encodings for different East Asian languages. This avoids the issues associated with unification, such as difficulty supporting multiple CJK languages with their associated character variants in a single document and font.
- Since ISO/IEC 2022 is a stateful encoding, a program cannot jump in the middle of a block of text to search, insert or delete characters. This makes manipulation of the text very cumbersome and slow when compared to non-stateful encodings. Any jump in the middle of the text may require a back up to the previous escape sequence before the bytes following the escape sequence can be interpreted.
- Due to the stateful nature of ISO/IEC 2022, an identical and equivalent character may be encoded in different character sets, which may be delegated to any of G0 through G3, which may be accessed using single shifts or by using locking shifts to GL or GR. Consequently, characters can be represented in multiple ways, meaning that two visually identical and equivalent strings can not be reliably compared for equality.
- Some systems, like DICOM and several e-mail clients, use a variant of ISO-2022 in addition to supporting several other encodings. This type of variation makes it difficult to portably transfer text between computer systems.
- UTF-1, the multi-byte Unicode transformation format compatible with ISO/IEC 2022, has various disadvantages in comparison with UTF-8, and switching from or to other charsets, as supported by ISO/IEC 2022, is typically unnecessary in Unicode documents.
- Because of its escape sequences, it is possible to construct attack byte sequences that round-trip from ISO/IEC 2022 to Unicode and back. Use of this encoding is thus treated as suspicious by malware protection suites.[better source needed]
- Concatenation can pose issues. Profiles such as ISO-2022-JP specify that the stream starts in the ASCII state and must end in the ASCII state. This is necessary to ensure that characters in concatenated ISO-2022-JP and/or ASCII streams will be interpreted in the correct set. However, it means that if a stream that ends in a multi-byte character is concatenated with one that starts with a multi-byte character, a pair of escape codes are generated switching to ASCII and immediately away from it. However, as stipulated in Unicode Technical Report #36 ("Unicode Security Considerations"), pairs of ISO 2022 escape sequences with no characters between them should generate a replacement character ("�") to prevent them from being used to mask malicious sequences. Implementing this measure, e.g. in Mozilla Thunderbird, has led to interoperability issues, with unexpected "�" characters being generated where two ISO-2022-JP streams have been concatenated.
- ISO 2709
- ISO/IEC 646
- C0 and C1 control codes
- MARC standards
- ISO/IEC JTC 1/SC 2
- "Standard ECMA 35" (PDF).
- RFC 1554 - ISO-2022-JP-2: Multilingual Extension of ISO-2022-JP. Tools.ietf.org. Retrieved on 2014-05-20.
- "KS X 1001:1992" (PDF).
- "KS C 5601:1987" (PDF). 1988-10-01.
- "DICOM ISO 2022 variation".
- RFC 1468
- Davis, Mark; Suignard, Michel (2014-09-19). "3.6.2 Some Output For All Input". Unicode Technical Report #36: Unicode Security Considerations (revision 15). Unicode Consortium.
- Sivonen, Henri (2018-12-17). "(UNSUBMITTED DRAFT) No U+FFFD Generation for Zero-Length ASCII-State Content between ISO-2022-JP Escape Sequences" (PDF).
- Lunde, Ken. CJKV Information Processing. Cambridge, Massachusetts: O'Reilly & Associates, 1998. ISBN 1-56592-224-7.
- ISO/IEC 2022:1994
- ISO/IEC 2022:1994/Cor 1:1999
- ECMA-35, equivalent to ISO/IEC 2022 and freely downloadable.
- International Register of Coded Character Sets to be Used with Escape Sequences, a full list of assigned character sets and their escape sequences
- History of Character Codes in North America, Europe, and East Asia from 1999, rev. 2004
- Ken Lunde's CJK.INF: a document on encoding Chinese, Japanese, and Korean (CJK) languages, including a discussion of the various variants of ISO/IEC 2022.