Misplaced Pages

UTF-16: Difference between revisions

Article snapshot taken from Wikipedia with creative commons attribution-sharealike license. Give it a read and then ask your questions in the chat. We can research this topic together.
Browse history interactively← Previous editContent deleted Content addedVisualWikitext
Revision as of 22:56, 28 August 2006 editWinBot (talk | contribs)39,167 editsm BOT - Unicodifying← Previous edit Latest revision as of 19:52, 25 December 2024 edit undoSpitzak (talk | contribs)Extended confirmed users9,994 edits Undid revision 1265204786 by 2607:FEA8:BB18:8F00:17EA:41E4:2897:EFFF (talk)Tag: Undo 
Line 1: Line 1:
{{Short description|Variable-width encoding of Unicode, using one or two 16-bit code units}}
{{Table Unicode}}
{{Infobox character encoding
In ], '''UTF-16''' is a variable-length (16 or 32 bits) ]. It is able to represent the complete ] ] with exactly two bytes and every other plane with exactly four bytes. '''UCS-2''' is a fixed-length (16 bits) subset of UTF-16, able to represent the basic multilingual plane only.
| name = UTF-16
| mime =
| alias =
| image = UTF-16 encoding.svg
| caption = Example of Unicode character encoding through UTF-16
| standard = Unicode Standard
| classification = ], ]
| lang = International
| status =
| encodes = ] (])
| extends = UCS-2
| prev =
| next =
}}
'''UTF-16''' (] ] Transformation Format) is a ] method capable of encoding all 1,112,064 valid ]s of Unicode.{{efn|This number of code points is a consequence of the design of UTF-16}} The encoding is ] as code points are encoded with one or two {{nobreak|16-bit}} ''code units''. UTF-16 arose from an earlier obsolete fixed-width 16-bit encoding now known as '''UCS-2''' (for 2-byte Universal Character Set),<ref name="unicode-6_0">{{Cite book |title=The Unicode Standard, version 6.0 |date=February 2011 |publisher=] |isbn=978-1-936213-01-6 |location=Mountain View, CA |pages=573 |chapter=C.2 Encoding Forms in ISO/IEC 10646 |quote= the term UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard. |chapter-url=https://www.unicode.org/versions/Unicode6.0.0/appC.pdf}}</ref><ref name="ucs-2-utf-16-differences">{{Cite web |title=FAQ: What is the difference between UCS-2 and UTF-16? |url=https://www.unicode.org/faq/utf_bom.html#utf16-11 |archive-url=https://web.archive.org/web/20030818043641/http://www.unicode.org/faq/basic_q.html#23 |archive-date=2003-08-18 |access-date=2024-03-19 |website=unicode.org |quote=UCS-2 is obsolete terminology which refers to a Unicode implementation up to Unicode 1.1 }}</ref> once it became clear that more than 2<sup>16</sup> (65,536) code points were needed,<ref name="Unicode.org/faq">{{cite web|title=What is UTF-16?|url=https://www.unicode.org/faq/utf_bom.html#utf16-1|website=The Unicode Consortium|publisher=Unicode, Inc.|quote=UTF-16 uses a single 16-bit code unit to encode over 60,000 of the most common characters in Unicode <!-- struck out: "most common 63K characters", and a pair of 16-bit code units, called surrogates, to encode the remainder of about 1 million struck out less commonly used characters in Unicode. Originally, Unicode was designed as a pure 16-bit encoding, aimed at representing all modern scripts. (Ancient scripts were to be represented with private-use characters.) Over time, and especially after the addition of over 14,500 composite characters for compatibility with legacy sets, it became clear that 16-bits were not sufficient for the user community. Out of this arose UTF-16.-->|access-date=7 January 2023}}</ref> including most emoji and important ] such as for personal and place names.<ref name="problems_of_only_BMP">{{Cite web |last=Lunde |first=Ken |date=2022-01-09 |title=2022 Top Ten List: Why Support Beyond-BMP Code Points? |url=https://ken-lunde.medium.com/2022-top-ten-list-why-support-beyond-bmp-code-points-6a946d7735f9 |website=Medium |language=en |quote=I first came up with the idea for this Top Ten List over 10 years ago, which was prompted by some environments that still supported only BMP code points. The idea, of course, was to motivate the developers of such environments to support code points beyond the BMP by providing an enumerated list of reasons to do so. And yes, there are still some environments that support only BMP code points, such as the VivaDesigner app.|access-date=2024-01-07}}</ref>


UTF-16 is used by the ] ], and many programming environments such as the ] and ]/ECMAScript. It is also sometimes used for ] and word-processing data files on Microsoft Windows. It is used by more modern implementations of ].<ref>{{cite web|url=https://www.twilio.com/engineering/2012/11/08/adventures-in-unicode-sms|title=Adventures in Unicode SMS|date=2012-11-08|publisher=Twilio|author=Chad Selph|access-date=2015-08-28|archive-url=https://web.archive.org/web/20150908104520/https://www.twilio.com/engineering/2012/11/08/adventures-in-unicode-sms|archive-date=2015-09-08|url-status=dead}}</ref> The variable length character of UTF-16, combined with the fact that most characters are ''not'' variable length (so variable length is rarely tested), has led to many bugs in software, including in Windows itself.<ref name=dialog_bug>{{Cite web |title=Should UTF-16 be considered harmful? |url=https://softwareengineering.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful |access-date=2024-11-20 |website=Software Engineering Stack Exchange |language=en |quote=File names editing in Window dialogs in broken (delete required 2 presses on backspace) }}</ref>
"UTF-16 is officially defined in Annex Q of ISO/IEC 10646-1. It is also described in ''The Unicode Standard'' version 3.0 and higher, as well as in the IETF's RFC 2781."
UTF-16 represents a character that has been assigned within the lower 65536 code points of Unicode or ISO/IEC 10646 as a '''single''' code value equivalent to the character's code point: 0 for 0, hexadecimal FFFD for FFFD, for example.


UTF-16 is the only encoding (still) allowed on the web that is incompatible with 8-bit ].<ref>{{Cite web|date=2020-06-10|title=HTML Living Standard|url=https://html.spec.whatwg.org/multipage/infrastructure.html#encoding-terminology|access-date=2020-06-15|website=w3.org|quote=<!--Since support for encodings that are not defined in Encoding is prohibited,--> UTF-16 encodings are the only encodings that this specification needs to treat as not being ASCII-compatible encodings.|archive-url=https://web.archive.org/web/20200908111027/https://html.spec.whatwg.org/multipage/infrastructure.html|archive-date=2020-09-08|url-status=deviated}}</ref>{{efn|UTF-32 is also incompatible with ASCII, but is not listed as a web-encoding.<ref>{{Cite web|url=https://encoding.spec.whatwg.org/|title=Encoding Standard|website=encoding.spec.whatwg.org|access-date=2023-04-22}}</ref>}} However it has never gained popularity on the web, where it is declared by under 0.003% of public web pages.<ref>{{Cite web|url=https://w3techs.com/technologies/details/en-utf16/all/all|title=Usage Statistics of UTF-16 for Websites, September 2024|website=w3techs.com|language=en|access-date=2024-09-03}}</ref> ], by comparison, accounts for over 98% of all web pages.<ref>{{Cite web|title=Usage Statistics of UTF-8 for Websites, September 2024|url=https://w3techs.com/technologies/details/en-utf8/all/all|access-date=2024-09-03|website=w3techs.com|language=en}}</ref> The ] considers UTF-8 "the mandatory encoding for all " and that for security reasons browser applications should not use UTF-16.<ref name="mandatory">{{Cite web|url=https://encoding.spec.whatwg.org/#security-background|title=Encoding Standard|website=encoding.spec.whatwg.org|quote=The UTF-8 encoding is the most appropriate encoding for interchange of Unicode, the universal coded character set. Therefore for new protocols and formats, as well as existing formats deployed in new contexts, this specification requires (and defines) the UTF-8 encoding. The problems outlined here go away when exclusively using UTF-8, which is one of the many reasons that UTF-8 is now the mandatory encoding for all text things on the Web.|language=en|access-date=2018-10-22}}</ref>
==Method for code points in Plane 1, Plane 2==
UTF-16 represents a character above hexadecimal FFFF using a pair of 16-bit words, known as a '''surrogate pair''', using "flag" code values from the range D800&ndash;DFFF. For example, the character at code point hexadecimal 10000 becomes the code value sequence D800 DC00, and the character at hexadecimal 10FFFD, the upper limit of Unicode, becomes the code value sequence DBFF DFFD. Unicode and ISO/IEC 10646 do not assign characters to any of the code points in the D800&ndash;DFFF range, so an individual code value from a surrogate pair does not ever represent a character.


== History ==
===Big-endian/Little-endian===
In the late 1980s, work began on developing a uniform encoding for a "Universal Character Set" (]) that would replace earlier language-specific encodings with one coordinated system. The goal was to include all required characters from most of the world's languages, as well as symbols from technical domains such as science, mathematics, and music. The original idea was to replace the typical 256-character encodings, which required 1 byte per character, with an encoding using 65,536 (2<sup>16</sup>) values, which would require 2 bytes (16 bits) per character.
These code values are then serialized as 16-bit words, one word per code value. Because the ] of these words varies according to the computer architecture, UTF-16 specifies three ''encoding schemes'': UTF-16, UTF-16LE, and UTF-16BE.


Two groups worked on this in parallel, ] and the ], the latter representing mostly manufacturers of computing equipment. The two groups attempted to synchronize their character assignments so that the developing encodings would be mutually compatible. The early 2-byte encoding was originally called "Unicode", but is now called "UCS-2".<ref name="unicode-6_0" /><ref name="ucs-2-utf-16-differences" /><ref name="mysql_UCS-2">{{cite web|url=https://dev.mysql.com/doc/refman/5.7/en/charset-unicode-ucs2.html|title=MySQL :: MySQL 5.7 Reference Manual :: 10.1.9.4 The ucs2 Character Set (UCS-2 Unicode Encoding)|website=dev.mysql.com}}</ref>
The ''UTF-16'' encoding scheme mandates that the byte order must be declared by prepending a ] before the first serialized character. This '''BOM''' is the encoded version of the Zero-Width No-Break Space character, Unicode number FEFF in hex, manifesting as the byte sequence FE FF for big-endian, or FF FE for little-endian. A BOM at the beginning of UTF-16 encoded data is considered to be a signature separate from the text itself; it is for the benefit of the decoder.


When it became increasingly clear that 2<sup>16</sup> characters would not suffice,<ref name="unicode.org/faq">{{cite web|title=What is UTF-16?|url=https://www.unicode.org/faq/utf_bom.html#utf16-1|website=The Unicode Consortium|publisher=Unicode, Inc.|access-date=29 March 2018}}</ref> ] introduced a larger 31-bit space and an encoding (]) that would require 4 bytes per character. This was resisted by the ], both because 4 bytes per character wasted a lot of memory and disk space, and because some manufacturers were already heavily invested in 2-byte-per-character technology. The UTF-16 encoding scheme was developed as a compromise and introduced with version 2.0 of the Unicode standard in July 1996.<ref>{{cite web |url=https://www.unicode.org/faq//utf_bom.html|title=Questions about encoding forms |access-date=2010-11-12}}</ref> It is fully specified in RFC 2781, published in 2000 by the ].<ref>ISO/IEC 10646:2014 "Information technology – Universal Coded Character Set (UCS)" sections 9 and 10.</ref><ref>''The Unicode Standard'' version 7.0 (2014) </ref>
The '''UTF-16LE''' and '''UTF-16BE''' encoding schemes are identical to the UTF-16 encoding scheme, but rather than using a BOM, the byte order is implicit in the name of the encoding (LE for little-endian, BE for big-endian). A BOM at the beginning of UTF-16LE or UTF-16BE encoded data is not considered to be a BOM; it is part of the text itself.


UTF-16 is specified in the latest versions of both the international standard ] and the Unicode Standard. "UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard."<ref name="unicode-6_0" /><ref name="ucs-2-utf-16-differences" /> UTF-16 will never be extended to support a larger number of code points or to support the code points that were replaced by surrogates, as this would violate the Unicode Stability Policy with respect to general category or surrogate code points.<ref>{{cite web|url=https://unicode.org/policies/stability_policy.html|title=Unicode Character Encoding Stability Policies|website=unicode.org}}</ref> (Any scheme that remains a ] would require allocating at least one ] (BMP) code point to start a sequence. Changing the purpose of a code point is disallowed.)
The ] has approved UTF-16, UTF-16BE, and UTF-16LE for use on the ], by those exact names (case insensitively). The aliases '''UTF_16''' or '''UTF16''' may be meaningful in some programming languages or software applications, but they are not standard names in Internet protocols.


== Description ==
==Major operating system usage==
Each Unicode ''code point'' is encoded either as one or two 16-bit ''code units''. Code points less than 2<sup>16</sup> ("in the BMP") are encoded with a single 16-bit code unit equal to the numerical value of the code point, as in the older UCS-2. Code points greater than or equal to 2<sup>16</sup> ("above the BMP") are encoded using ''two'' 16-bit code units. These two 16-bit code units are chosen from the ] {{tt|0xD800–0xDFFF}} which had not previously been assigned to characters. Values in this range are not used as characters, and UTF-16 provides no legal way to code them as individual code points. A UTF-16 stream, therefore, consists of single 16-bit codes outside the surrogate range, and pairs of 16-bit values that are within the surrogate range.
UTF-16 is the native internal representation of text in the ]/]/]/], ], and ] operating systems; the ] and ] bytecode environments; ]'s Cocoa and Core Foundation frameworks; and the ] cross-platform graphical widget toolkit.


=== U+0000 to U+D7FF and U+E000 to U+FFFF ===
==UCS-2==
{{hatnote|U+D800 to U+DFFF have a special purpose, see below.}}
'''UCS-2''' is an obsolete specification that can only encode the ], and consequently has been superseded by UTF-16. In some materials, the terms UCS-2 and UTF-16 are erroneously conflated and used interchangeably; UTF-16 is often mislabeled UCS-2.
Both UTF-16 and UCS-2 encode code points in this range as single 16-bit code units that are numerically equal to the corresponding code points. These code points in the ] (BMP) are the ''only'' code points that can be represented in UCS-2.{{Citation needed|date=November 2015}} As of Unicode 9.0, some modern non-Latin Asian, Middle-Eastern, and African scripts fall outside this range, as do most ] characters.


=== Code points from U+010000 to U+10FFFF ===
] systems prior to ] only support UCS-2.
Code points from the other ] are encoded as two 16-bit ''code units'' called a ''surrogate pair''. The first code unit is a ''high surrogate'' and the second is a ''low surrogate'' (These are also known as "leading" and "trailing" surrogates, respectively, analogous to the leading and trailing bytes of UTF-8.<ref name="Unicode7Ch3s8">{{cite book |title=The Unicode Standard, Version 7.0—Core Specification |editor1-last=Allen |editor1-first=Julie D. |editor2-last=Anderson |editor2-first=Deborah |editor3-last=Becker |editor3-first=Joe |editor3-link=Joe Becker (Unicode) |editor4-last=Cook |editor4-first=Richard |publisher=] |date=2014 |place=Mountain View |url=https://www.unicode.org/versions/Unicode7.0.0/ |section=3.8 Surrogates |page=118 |chapter-url=https://www.unicode.org/versions/Unicode7.0.0/ch03.pdf |archive-url=https://ghostarchive.org/archive/20221009/https://www.unicode.org/versions/Unicode7.0.0/ch03.pdf |archive-date=2022-10-09 |url-status=live |access-date=3 November 2014 }}</ref>):


{| class="wikitable floatright" style="text-align: center; font-family:monospace"
==Examples==
|+ UTF-16 decoder
!{{diagonal split header|<span style{{=}}"color:#920000;">High</span> | <span style{{=}}"color:#000092;">Low</span>}}
!<span style="color:#000092;">DC00</span>
!<span style="color:#000092;">DC01</span>
!<span style="color:#000092;">&nbsp;&nbsp;&nbsp;...&nbsp;&nbsp;&nbsp;</span>
!<span style="color:#000092;">DFFF</span>
|-
!<span style="color:#920000;">D800</span>
|010000||010001||...||0103FF
|-
!<span style="color:#920000;">D801</span>
|010400||010401||...||0107FF
|-align="center"
!<span style="color:#920000;">⋮</span>
|⋮||⋮||⋱||⋮
|-
!<span style="color:#920000;">DBFF</span>
|10FC00||10FC01||...||10FFFF
|}


* 0x10000 is subtracted from the code point ''(U)'', leaving a 20-bit number ''(U')'' in the hex number range 0x00000–0xFFFFF.
{| class="wikitable"
* The high ten bits (in the range 0x000–0x3FF) are added to 0xD800 to give the first 16-bit ''code unit'' or ''high surrogate'' ''(W1)'', which will be in the range {{color|#920000|0xD800–0xDBFF}}.
|-
* The low ten bits (also in the range 0x000–0x3FF) are added to 0xDC00 to give the second 16-bit ''code unit'' or ''low surrogate'' ''(W2)'', which will be in the range {{color|#000092|0xDC00–0xDFFF}}.
! code point

! character
Illustrated visually, the distribution of ''U''' between ''W1'' and ''W2'' looks like:<ref>{{Cite journal|url=https://tools.ietf.org/html/rfc2781.html|title=UTF-16, an encoding of ISO 10646|last1=Yergeau|first1=Francois|last2=Hoffman|first2=Paul|website=tools.ietf.org|date=February 2000|language=en|access-date=2019-06-18}}</ref><syntaxhighlight lang="text">
! UTF-16 code value(s)
U' = yyyyyyyyyyxxxxxxxxxx // U - 0x10000
! glyph*
W1 = 110110yyyyyyyyyy // 0xD800 + yyyyyyyyyy
|-
W2 = 110111xxxxxxxxxx // 0xDC00 + xxxxxxxxxx
| 122 (hex 7A)
</syntaxhighlight>
| small Z (Latin)
Since the ranges for the ''high surrogates'' ({{color|#920000|0xD800–0xDBFF}}), ''low surrogates'' ({{color|#000092|0xDC00–0xDFFF}}), and valid BMP characters (0x0000–0xD7FF, 0xE000–0xFFFF) are ], it is not possible for a surrogate to match a BMP character, or for two adjacent ''code units'' to look like a legal ''surrogate pair''. This simplifies searches a great deal. It also means that UTF-16 is '']'' on 16-bit words: whether a code unit starts a character can be determined without examining earlier code units (i.e. the type of ''code unit'' can be determined by the ranges of values in which it falls). UTF-8 shares these advantages, but many earlier multi-byte encoding schemes (such as ] and other Asian multi-byte encodings) did not allow unambiguous searching and could only be synchronized by re-parsing from the start of the string. UTF-16 is not self-synchronizing if one byte is lost or if traversal starts at a random byte.
| 007A

| z
Because the most commonly used characters are all in the BMP, handling of surrogate pairs is often not thoroughly tested. This leads to persistent bugs and potential security holes, even in popular and well-reviewed application software (e.g. {{CVE|2008-2938|2012-2135}}).
|-

| 27700 (hex 6C34)
=== U+D800 to U+DFFF (surrogates) ===
| water (Chinese)
{{Refimprove section|date=August 2023}}
| 6C34
The official Unicode standard says that no UTF forms, including UTF-16, can encode the surrogate code points. Since these will never be assigned a character, there should be no reason to encode them. However, Windows allows unpaired surrogates in filenames<ref>{{cite web |title=Maximum Path Length Limitation |url=https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation |website=Microsoft | date=2022-07-18 |access-date=2022-10-10 |quote= the file system treats path and file names as an opaque sequence of WCHARs}}</ref> and other places, which generally means they have to be supported by software in spite of their exclusion from the Unicode standard.
| 水

|-
UCS-2, UTF-8, and ] can encode these code points in trivial and obvious ways, and a large amount of software does so, even though the standard states that such arrangements should be treated as encoding errors.
| 119070 (hex 1D11E)

| ]
It is possible to unambiguously encode an ''unpaired surrogate'' (a high surrogate code point not followed by a low one, or a low one not preceded by a high one) in the format of UTF-16 by using a code unit equal to the code point. The result is not valid UTF-16, but the majority of UTF-16 encoder and decoder implementations do this when translating between encodings.{{citation needed|date=October 2011}}<!-- Python 2.6 decode of UTF16 does this on Linux, and it correctly handles surrogate pairs. All "CESU" decoders do it too, though they also mis-translate correct surrogate pairs into 2 characters. -->
| D834 DD1E <!--this has been repeatedly changed to D874 by people forgetting about the subtraction step. Please don't do it again54. --><!--...em, is there a font that has the musical G clef? question by user:abdull) --><!--code 2001 probably has it but note that if you are on windows you need to do special setup to use non bmp characters i think the info is on the talk page of the code 2000 article or something like that-->

| 턞
=== Examples ===
|}
To encode U+10437 (𐐷) to UTF-16:

* Subtract 0x10000 from the code point, leaving 0x0437.
* For the high surrogate, shift right by 10 (divide by 0x400), then add 0xD800, resulting in 0x0001 + 0xD800 = 0xD801.
* For the low surrogate, take the low 10 bits (remainder of dividing by 0x400), then add 0xDC00, resulting in 0x0037 + 0xDC00 = 0xDC37.

To decode U+10437 (𐐷) from UTF-16:

* Take the high surrogate (0xD801) and subtract 0xD800, then multiply by 0x400, resulting in 0x0001 × 0x400 = 0x0400.
* Take the low surrogate (0xDC37) and subtract 0xDC00, resulting in 0x37.
* Add these two results together (0x0437), and finally add 0x10000 to get the final code point, 0x10437.

The following table summarizes this conversion, as well as others. The colors indicate how bits from the code point are distributed among the UTF-16 bytes. Additional bits added by the UTF-16 encoding process are shown in black.


{| class="wikitable" {| class="wikitable"
|- |-
!colspan=2|Character
! colspan="3" | "水z턞" (water, z, G clef), UTF-16 encoded
! Binary code point
! Binary UTF-16
! UTF-16 hex<br />code units
! UTF-16BE<br />hex bytes
! UTF-16LE<br />hex bytes
|- |-
|] || <code>U+0024</code>
! labeled encoding
|align=right|<code>{{Font color|#000092|0000 0000 0010 0100}}</code>
! byte order
|align=right|<code>{{Font color|#000092|0000 0000 0010 0100}}</code>
! byte sequence
|align=right|<code>{{Font color|#000092|0024}}</code>
|align=right|<code>{{Font color|#000092|00 24}}</code>
|align=right|<code>{{Font color|#000092|24 00}}</code>
|- |-
|] || <code>U+20AC</code>
| UTF-16LE
|align=right|<code>{{Font color|#000092|0010 0000 1010 1100}}</code>
| little-endian
|align=right|<code>{{Font color|#000092|0010 0000 1010 1100}}</code>
| 34 6C, 7A 00, 34 D8 1E DD
|align=right|<code>{{Font color|#000092|20AC}}</code>
|align=right|<code>{{Font color|#000092|20 AC}}</code>
|align=right|<code>{{Font color|#000092|AC 20}}</code>
|- |-
|] || <code>U+10437</code>
| UTF-16BE
|align=right|<code>{{Font color|#920000|0001 0000 01}}{{Font color|#000092|00 0011 0111}}</code>
| big-endian
|align=right|<code>1101 10{{Font color|#920000|00 0000 0001}} 1101 11{{Font color|#000092|00 0011 0111}}</code>
| 6C 34, 00 7A, D8 34 DD 1E
|align=right|<code>{{Font color|#920000|D801}} {{Font color|#000092|DC37}}</code>
|align=right|<code>{{Font color|#920000|D8 01}} {{Font color|#000092|DC 37}}</code>
|align=right|<code>{{Font color|#920000|01 D8}} {{Font color|#000092|37 DC}}</code>
|- |-
|] || <code>U+24B62</code>
| UTF-16
|align=right|<code>{{Font color|#920000|0010 0100 10}}{{Font color|#000092|11 0110 0010}}</code>
| little-endian, with BOM
|align=right|<code>1101 10{{Font color|#920000|00 0101 0010}} 1101 11{{Font color|#000092|11 0110 0010}}</code>
| FF FE, 34 6C, 7A 00, 34 D8 1E DD
|align=right|<code>{{Font color|#920000|D852}} {{Font color|#000092|DF62}}</code>
|-
|align=right|<code>{{Font color|#920000|D8 52}} {{Font color|#000092|DF 62}}</code>
| UTF-16
|align=right|<code>{{Font color|#920000|52 D8}} {{Font color|#000092|62 DF}}</code>
| big-endian, with BOM
| FE FF, 6C 34, 00 7A, D8 34 DD 1E
|} |}


{{anchor|UTF-16LE|UTF-16BE}}
''* Appropriate font and software are required to see the correct glyphs.''
== Byte-order encoding schemes ==
UTF-16 and UCS-2 produce a sequence of 16-bit code units. Since most communication and storage protocols are defined for bytes, and each unit thus takes two 8-bit bytes, the order of the bytes may depend on the ] (byte order) of the computer architecture.


To assist in recognizing the byte order of code units, '''UTF-16''' allows a ] (BOM), a code point with the value U+FEFF, to precede the first actual coded value.{{efn|UTF-8 encoding produces byte values strictly less than 0xFE, so either byte in the BOM sequence also identifies the encoding as UTF-16 (assuming that UTF-32 is not expected).}} (U+FEFF is the invisible ]/ZWNBSP character).{{efn|Use of U+FEFF as the character ZWNBSP instead of as a BOM has been deprecated in favor of U+2060 (WORD JOINER); see at Unicode.org. But if an application interprets an initial BOM as a character, the ZWNBSP character is invisible, so the impact is minimal.}} If the endian architecture of the decoder matches that of the encoder, the decoder detects the 0xFEFF value, but an opposite-endian decoder interprets the BOM as the ] value U+FFFE reserved for this purpose. This incorrect result provides a hint to perform byte-swapping for the remaining values.
===Example UTF-16 Encoding Procedure===


If the BOM is missing, RFC 2781 recommends{{efn|{{IETF RFC|2781}} section 4.3 says that if there is no BOM, "the text SHOULD be interpreted as being big-endian." According to section 1.2, the meaning of the term "SHOULD" is governed by {{IETF RFC|2119}}. In that document, section 3 says "... there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course".}} that big-endian (BE) encoding be assumed. In practice, due to Windows using little-endian (LE) order by default, many applications assume little-endian encoding. It is also reliable to detect endianness by looking for null bytes, on the assumption that characters less than U+0100 are very common. If more even bytes (starting at 0) are null, then it is big-endian.
The character at code point U+64321 (hexadecimal) is to be encoded in UTF-16. Since it is above U+FFFF, it must be encoded with a surrogate pair, as follows:


The standard also allows the byte order to be stated explicitly by specifying '''UTF-16BE''' or '''UTF-16LE''' as the encoding type. When the byte order is specified explicitly this way, a BOM is specifically ''not'' supposed to be prepended to the text, and a U+FEFF at the beginning should be handled as a ZWNBSP character. Most applications ignore a BOM in all cases despite this rule.
v = 0x64321
v&prime; = v - 0x10000
= 0x54321
= 0101 0100 0011 0010 0001
vh = 0101010000 // higher 10 bits of v&prime;
vl = 1100100001 // lower 10 bits of v&prime;
w1 = 0xD800 // the resulting 1st word is initialized with the lower bracket
w2 = 0xDC00 // the resulting 2nd word is initialized with the higher bracket
w1 = w1 | vh
= 1101 1000 0000 0000 | 01 0101 0000
= 1101 1001 0101 0000
= 0xD950
w2 = w2 | vl
= 1101 1100 0000 0000 | 11 0010 0001
= 1101 1111 0010 0001
= 0xDF21


For ] protocols, ] has approved "UTF-16", "UTF-16BE", and "UTF-16LE" as the names for these encodings (the names are case insensitive). The aliases '''UTF_16''' or '''UTF16''' may be meaningful in some programming languages or software applications, but they are not standard names in Internet protocols.
The correct UTF-16 encoding for this character is thus the following word sequence:
0xD950 0xDF21


Similar designations, '''UCS-2BE''' and '''UCS-2LE''', are used to show versions of '''UCS-2'''.
Since the character is above U+FFFF, the character cannot be encoded in UCS-2.

== Size ==
A "character" may use any number of Unicode code points.<ref name="extended grapheme">{{Cite web|title=It's not wrong that "🤦🏼‍♂️".length == 7|url=https://hsivonen.fi/string-length/|access-date=2021-03-15|website=hsivonen.fi}}</ref> For instance an ] takes 8 bytes, since it is "constructed from a pair of Unicode scalar values"<ref>{{Cite web|title=Apple Developer Documentation|url=https://developer.apple.com/documentation/swift/string|access-date=2021-03-15|website=developer.apple.com}}</ref> (and those values are outside the BMP and require 4 bytes each). UTF-16 in no way assists in "counting characters" or in "measuring the width of a string".

UTF-16 is often claimed to be more space-efficient than ] for East Asian languages, since it uses two bytes for characters that take 3 bytes in UTF-8. Since real text contains many spaces, numbers, punctuation, markup (for e.g. web pages), and control characters, which take only one byte in UTF-8, this is only true for artificially constructed dense blocks of text.{{citation needed|date=November 2022}} A more serious claim can be made for ] and ], which use multi-letter words and all the letters take 3 bytes in UTF-8 and only 2 in UTF-16.

In addition the Chinese Unicode encoding standard ] always produces files the same size or smaller than UTF-16 for all languages, not just for Chinese (it does this by sacrificing self-synchronization).

== Usage ==
UTF-16 is used for text in the OS&nbsp;] of all currently supported versions of ] (and including at least all since ]/]/]/]/]/]<ref>. Retrieved 2011-03-08 "These functions use UTF-16 (wide character) encoding (…) used for native Unicode encoding on Windows operating systems."</ref>) including ]. In Windows XP, no code point above U+FFFF is included in any font delivered with Windows for European languages.<ref>{{cite web |title=Unicode|publisher=microsoft.com |url=https://msdn.microsoft.com/en-us/library/dd374081.aspx |access-date=2009-07-20}}</ref><ref>{{cite web |title=Surrogates and Supplementary Characters |publisher=microsoft.com |url=https://msdn.microsoft.com/en-us/library/dd374069.aspx |access-date=2009-07-20}}</ref> Older ] systems (prior to Windows 2000) only support ].<ref>{{cite web |title=Description of storing UTF-8 data in SQL Server |publisher=microsoft.com |date=7 December 2005 |url=https://support.microsoft.com/kb/232580 |access-date=2008-02-01}}</ref> Files and network data tend to be a mix of UTF-16, UTF-8, and legacy byte encodings.

While there's been some UTF-8 support for even Windows XP,<ref>{{Cite web|title= Patch for cmd.exe for windows xp for cp 65001 - Page 2 - DosTips.com|url=https://www.dostips.com/forum/viewtopic.php?t=5588&start=15|access-date=2021-06-17|website=www.dostips.com}}</ref> it was ] (in particular the ability to name a file using UTF-8) in Windows 10 ] 17035 and the May 2019 update. As of May 2019, ], on Windows and ], instead of other 8-bit encodings.<ref name="Microsoft-UTF-8">{{cite web|title=Use UTF-8 code pages in Windows apps|url=https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page |access-date=2020-06-06 |quote=As of Windows version&nbsp;1903 (May&nbsp;2019 update), you can use the ActiveCodePage property in the appxmanifest for packaged apps, or the fusion manifest for unpackaged apps, to force a process to use UTF-8 as the process code page. <code>CP_ACP</code> equates to <code>CP_UTF8</code> only if running on Windows version&nbsp;1903 (May&nbsp;2019 update) or above and the ActiveCodePage property described above is set to UTF-8. Otherwise, it honors the legacy system code page. We recommend using <code>CP_UTF8</code> explicitly. |website=learn.microsoft.com |language=en-us}}</ref> It is unclear if they are recommending usage of UTF-8 over UTF-16, though they do state "UTF-16 is a unique burden that Windows places on code that targets multiple platforms."<ref name="Microsoft GDK">{{Cite web |title=UTF-8 support in the Microsoft Game Development Kit (GDK) - Microsoft Game Development Kit |url=https://learn.microsoft.com/en-us/gaming/gdk/_content/gc/system/overviews/utf-8 |access-date=2023-03-05 |website=learn.microsoft.com |language=en-us |quote=By operating in UTF-8, you can ensure maximum compatibility Windows operates natively in UTF-16 (or WCHAR), which requires code page conversions by using MultiByteToWideChar and WideCharToMultiByte. This is a unique burden that Windows places on code that targets multiple platforms. The Microsoft Game Development Kit (GDK) and Windows in general are moving forward to support UTF-8 to remove this unique burden of Windows on code targeting or interchanging with multiple platforms and the web. Also, this results in fewer internationalization issues in apps and games and reduces the test matrix that's required to get it right.}}</ref>

The ] operating system designates ] (]) 13488 for UCS-2 encoding and CCSID 1200 for UTF-16 encoding, though the system treats them both as UTF-16.<ref>{{cite web |url=https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_74/nls/rbagsucs2.htm |title=UCS-2 and its relationship to Unicode (UTF-16) |publisher=] |access-date=2019-04-26}}</ref>

UTF-16 is used by the ] operating systems; the ] environments; and the ] cross-platform graphical ].

] used in Nokia S60 handsets and Sony Ericsson ] handsets uses UCS-2. ] handsets use UTF-16 for ] instead of UCS-2 described in the ] (]) and IS-637 (]) standards.<ref>{{cite web|url=https://www.twilio.com/engineering/2012/11/08/adventures-in-unicode-sms|title=Adventures in Unicode SMS|author-last=Selph|author-first=Chad|date=2012-11-08|publisher=Twilio|archive-url=https://web.archive.org/web/20121109052626/https://www.twilio.com/engineering/2012/11/08/adventures-in-unicode-sms|access-date=2015-08-28|archive-date=2012-11-09}}</ref>

The ], used in ] media, encodes file names using UCS-2BE (up to sixty-four Unicode characters per file name).

] version 2.0 officially only used UCS-2 internally, but the UTF-8 decoder to "Unicode" produced correct UTF-16. There was also the ability to compile Python so that it used UTF-32 internally, this was sometimes done on Unix. Python 3.3 switched internal storage to use one of ], UCS-2, or UTF-32 depending on the largest code point in the string.<ref>{{cite web |url=https://www.python.org/dev/peps/pep-0393/ |title=PEP 0393 – Flexible String Representation |work=Python.org |access-date=2015-05-29}}</ref> Python 3.12 drops some functionality (for CPython extensions) to make it easier to migrate to ] for all strings.<ref>{{Cite web |title=PEP 623 – Remove wstr from Unicode {{!}} peps.python.org |url=https://peps.python.org/pep-0623/ |access-date=2023-02-24 |website=peps.python.org}}</ref>

] originally used UCS-2, and added UTF-16 supplementary character support in ]. Recently they have encouraged dumping support for any 8-bit encoding other than UTF-8<ref name="Java">{{Cite web |title=JEP 400: UTF-8 by Default |url=https://openjdk.org/jeps/400 |access-date=2023-03-12 |website=openjdk.org}}</ref> but internally UTF-16 is still used.

] may use UCS-2 or UTF-16.<ref name="mathiasbynens.be">{{cite web|url=https://mathiasbynens.be/notes/javascript-encoding|title=JavaScript's internal character encoding: UCS-2 or UTF-16? · Mathias Bynens}}</ref> As of ES2015, string methods and regular expression flags have been added to the language that permit handling strings from an encoding-agnostic perspective.

] uses UTF-16 to encode strings by default.

], Apple's preferred application language, used UTF-16 to store strings until version 5 which switched to UTF-8.<ref>{{Cite web|date=2019-03-20|title=UTF-8 String|url=https://swift.org/blog/utf8-string/|access-date=2020-08-20|website=Swift.org|language=en}}</ref>

Quite a few languages make the encoding part of the string object, and thus store and support a large set of encodings including UTF-16. Most consider UTF-16 and UCS-2 to be different encodings. Examples are the ] language<ref>{{cite web|url=https://php.net/manual/en/mbstring.supported-encodings.php|title=PHP: Supported Character Encodings - Manual|website=php.net}}</ref> and ].<ref>{{Cite web |title=MySQL :: MySQL 8.0 Reference Manual :: 10.9.2 The utf8mb3 Character Set (3-Byte UTF-8 Unicode Encoding) |url=https://dev.mysql.com/doc/refman/8.0/en/charset-unicode-utf8mb3.html |access-date=2023-02-24 |website=dev.mysql.com}}</ref>

A method to determine what encoding a system is using internally is to ask for the "length" of string containing a single non-BMP character. If the length is 2 then UTF-16 is being used. 4 indicates UTF-8. 3 or 6 may indicate ]. 1 ''may'' indicate UTF-32, but more likely indicates the language decodes the string to code points before measuring the "length".

In many languages, quoted strings need a new syntax for quoting non-BMP characters, as the C-style <code>"\uXXXX"</code> syntax explicitly limits itself to 4 hex digits. The following examples illustrate the syntax for the non-BMP character {{unichar|1D11E|MUSICAL SYMBOL G CLEF}}:
* The most common (], ], ], and several other languages) is an upper-case 'U' with ''8'' hex digits such as <code>"\U0001D11E"</code>.<ref>{{cite web
|url=http://en.csharp-online.net/ECMA-334:_9.4.1_Unicode_escape_sequences
|title=ECMA-334: 9.4.1 Unicode escape sequences|website=en.csharp-online.net
|archive-url=https://web.archive.org/web/20130215065218/http://en.csharp-online.net/ECMA-334:_9.4.1_Unicode_escape_sequences
|archive-date=2013-02-15}}</ref>
* Java 7 regular expressions, ], and Perl, use <code>"\x{1D11E}"</code>.
* ] 2015 (JavaScript) uses <code>"\u{1D11E}"</code>.
* In many other cases (such as Java outside of regular expressions),<ref>''Lexical Structure: Unicode Escapes'' in {{cite web |url=https://docs.oracle.com/javase/specs/jls/se6/html/lexical.html#3.3 |title=The Java Language Specification, Third Edition|website = Sun Microsystems, Inc. |year = 2005 | access-date=2019-10-11}}</ref> the only way to get non-BMP characters is to enter the surrogate halves individually: <code>"\uD834\uDD1E"</code>.


== See also == == See also ==
* ] * ]
* ]
* ] * ]
* ]
* ]

== Notes ==
{{notelist}}

== References ==
{{reflist|30em}}


==External links== == External links ==
* *
* *
* *
* *
* {{IETF RFC|2781|link=no}}: UTF-16, an encoding of ISO 10646
*


] {{Unicode navigation}}
] {{Character encoding}}


] ]
]
]
]
]
]
]
]
]
]
]
]
]
]
]
]

Latest revision as of 19:52, 25 December 2024

Variable-width encoding of Unicode, using one or two 16-bit code units
UTF-16
Example of Unicode character encoding through UTF-16
Language(s)International
StandardUnicode Standard
ClassificationUnicode Transformation Format, variable-width encoding
ExtendsUCS-2
Transforms / EncodesISO/IEC 10646 (Unicode)

UTF-16 (16-bit Unicode Transformation Format) is a character encoding method capable of encoding all 1,112,064 valid code points of Unicode. The encoding is variable-length as code points are encoded with one or two 16-bit code units. UTF-16 arose from an earlier obsolete fixed-width 16-bit encoding now known as UCS-2 (for 2-byte Universal Character Set), once it became clear that more than 2 (65,536) code points were needed, including most emoji and important CJK characters such as for personal and place names.

UTF-16 is used by the Microsoft Windows API, and many programming environments such as the Java programming language and JavaScript/ECMAScript. It is also sometimes used for plain text and word-processing data files on Microsoft Windows. It is used by more modern implementations of SMS. The variable length character of UTF-16, combined with the fact that most characters are not variable length (so variable length is rarely tested), has led to many bugs in software, including in Windows itself.

UTF-16 is the only encoding (still) allowed on the web that is incompatible with 8-bit ASCII. However it has never gained popularity on the web, where it is declared by under 0.003% of public web pages. UTF-8, by comparison, accounts for over 98% of all web pages. The Web Hypertext Application Technology Working Group (WHATWG) considers UTF-8 "the mandatory encoding for all " and that for security reasons browser applications should not use UTF-16.

History

In the late 1980s, work began on developing a uniform encoding for a "Universal Character Set" (UCS) that would replace earlier language-specific encodings with one coordinated system. The goal was to include all required characters from most of the world's languages, as well as symbols from technical domains such as science, mathematics, and music. The original idea was to replace the typical 256-character encodings, which required 1 byte per character, with an encoding using 65,536 (2) values, which would require 2 bytes (16 bits) per character.

Two groups worked on this in parallel, ISO/IEC JTC 1/SC 2 and the Unicode Consortium, the latter representing mostly manufacturers of computing equipment. The two groups attempted to synchronize their character assignments so that the developing encodings would be mutually compatible. The early 2-byte encoding was originally called "Unicode", but is now called "UCS-2".

When it became increasingly clear that 2 characters would not suffice, IEEE introduced a larger 31-bit space and an encoding (UCS-4) that would require 4 bytes per character. This was resisted by the Unicode Consortium, both because 4 bytes per character wasted a lot of memory and disk space, and because some manufacturers were already heavily invested in 2-byte-per-character technology. The UTF-16 encoding scheme was developed as a compromise and introduced with version 2.0 of the Unicode standard in July 1996. It is fully specified in RFC 2781, published in 2000 by the IETF.

UTF-16 is specified in the latest versions of both the international standard ISO/IEC 10646 and the Unicode Standard. "UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard." UTF-16 will never be extended to support a larger number of code points or to support the code points that were replaced by surrogates, as this would violate the Unicode Stability Policy with respect to general category or surrogate code points. (Any scheme that remains a self-synchronizing code would require allocating at least one Basic Multilingual Plane (BMP) code point to start a sequence. Changing the purpose of a code point is disallowed.)

Description

Each Unicode code point is encoded either as one or two 16-bit code units. Code points less than 2 ("in the BMP") are encoded with a single 16-bit code unit equal to the numerical value of the code point, as in the older UCS-2. Code points greater than or equal to 2 ("above the BMP") are encoded using two 16-bit code units. These two 16-bit code units are chosen from the UTF-16 surrogate range 0xD800–0xDFFF which had not previously been assigned to characters. Values in this range are not used as characters, and UTF-16 provides no legal way to code them as individual code points. A UTF-16 stream, therefore, consists of single 16-bit codes outside the surrogate range, and pairs of 16-bit values that are within the surrogate range.

U+0000 to U+D7FF and U+E000 to U+FFFF

U+D800 to U+DFFF have a special purpose, see below.

Both UTF-16 and UCS-2 encode code points in this range as single 16-bit code units that are numerically equal to the corresponding code points. These code points in the Basic Multilingual Plane (BMP) are the only code points that can be represented in UCS-2. As of Unicode 9.0, some modern non-Latin Asian, Middle-Eastern, and African scripts fall outside this range, as do most emoji characters.

Code points from U+010000 to U+10FFFF

Code points from the other planes are encoded as two 16-bit code units called a surrogate pair. The first code unit is a high surrogate and the second is a low surrogate (These are also known as "leading" and "trailing" surrogates, respectively, analogous to the leading and trailing bytes of UTF-8.):

UTF-16 decoder
LowHigh DC00 DC01    ...    DFFF
D800 010000 010001 ... 0103FF
D801 010400 010401 ... 0107FF
DBFF 10FC00 10FC01 ... 10FFFF
  • 0x10000 is subtracted from the code point (U), leaving a 20-bit number (U') in the hex number range 0x00000–0xFFFFF.
  • The high ten bits (in the range 0x000–0x3FF) are added to 0xD800 to give the first 16-bit code unit or high surrogate (W1), which will be in the range 0xD800–0xDBFF.
  • The low ten bits (also in the range 0x000–0x3FF) are added to 0xDC00 to give the second 16-bit code unit or low surrogate (W2), which will be in the range 0xDC00–0xDFFF.

Illustrated visually, the distribution of U' between W1 and W2 looks like:

U' = yyyyyyyyyyxxxxxxxxxx  // U - 0x10000
W1 = 110110yyyyyyyyyy      // 0xD800 + yyyyyyyyyy
W2 = 110111xxxxxxxxxx      // 0xDC00 + xxxxxxxxxx

Since the ranges for the high surrogates (0xD800–0xDBFF), low surrogates (0xDC00–0xDFFF), and valid BMP characters (0x0000–0xD7FF, 0xE000–0xFFFF) are disjoint, it is not possible for a surrogate to match a BMP character, or for two adjacent code units to look like a legal surrogate pair. This simplifies searches a great deal. It also means that UTF-16 is self-synchronizing on 16-bit words: whether a code unit starts a character can be determined without examining earlier code units (i.e. the type of code unit can be determined by the ranges of values in which it falls). UTF-8 shares these advantages, but many earlier multi-byte encoding schemes (such as Shift JIS and other Asian multi-byte encodings) did not allow unambiguous searching and could only be synchronized by re-parsing from the start of the string. UTF-16 is not self-synchronizing if one byte is lost or if traversal starts at a random byte.

Because the most commonly used characters are all in the BMP, handling of surrogate pairs is often not thoroughly tested. This leads to persistent bugs and potential security holes, even in popular and well-reviewed application software (e.g. CVE-2008-2938, CVE-2012-2135).

U+D800 to U+DFFF (surrogates)

This section needs additional citations for verification. Please help improve this article by adding citations to reliable sources in this section. Unsourced material may be challenged and removed. (August 2023) (Learn how and when to remove this message)

The official Unicode standard says that no UTF forms, including UTF-16, can encode the surrogate code points. Since these will never be assigned a character, there should be no reason to encode them. However, Windows allows unpaired surrogates in filenames and other places, which generally means they have to be supported by software in spite of their exclusion from the Unicode standard.

UCS-2, UTF-8, and UTF-32 can encode these code points in trivial and obvious ways, and a large amount of software does so, even though the standard states that such arrangements should be treated as encoding errors.

It is possible to unambiguously encode an unpaired surrogate (a high surrogate code point not followed by a low one, or a low one not preceded by a high one) in the format of UTF-16 by using a code unit equal to the code point. The result is not valid UTF-16, but the majority of UTF-16 encoder and decoder implementations do this when translating between encodings.

Examples

To encode U+10437 (𐐷) to UTF-16:

  • Subtract 0x10000 from the code point, leaving 0x0437.
  • For the high surrogate, shift right by 10 (divide by 0x400), then add 0xD800, resulting in 0x0001 + 0xD800 = 0xD801.
  • For the low surrogate, take the low 10 bits (remainder of dividing by 0x400), then add 0xDC00, resulting in 0x0037 + 0xDC00 = 0xDC37.

To decode U+10437 (𐐷) from UTF-16:

  • Take the high surrogate (0xD801) and subtract 0xD800, then multiply by 0x400, resulting in 0x0001 × 0x400 = 0x0400.
  • Take the low surrogate (0xDC37) and subtract 0xDC00, resulting in 0x37.
  • Add these two results together (0x0437), and finally add 0x10000 to get the final code point, 0x10437.

The following table summarizes this conversion, as well as others. The colors indicate how bits from the code point are distributed among the UTF-16 bytes. Additional bits added by the UTF-16 encoding process are shown in black.

Character Binary code point Binary UTF-16 UTF-16 hex
code units
UTF-16BE
hex bytes
UTF-16LE
hex bytes
$ U+0024 0000 0000 0010 0100 0000 0000 0010 0100 0024 00 24 24 00
U+20AC 0010 0000 1010 1100 0010 0000 1010 1100 20AC 20 AC AC 20
𐐷 U+10437 0001 0000 0100 0011 0111 1101 1000 0000 0001 1101 1100 0011 0111 D801 DC37 D8 01 DC 37 01 D8 37 DC
𤭢 U+24B62 0010 0100 1011 0110 0010 1101 1000 0101 0010 1101 1111 0110 0010 D852 DF62 D8 52 DF 62 52 D8 62 DF

Byte-order encoding schemes

UTF-16 and UCS-2 produce a sequence of 16-bit code units. Since most communication and storage protocols are defined for bytes, and each unit thus takes two 8-bit bytes, the order of the bytes may depend on the endianness (byte order) of the computer architecture.

To assist in recognizing the byte order of code units, UTF-16 allows a byte order mark (BOM), a code point with the value U+FEFF, to precede the first actual coded value. (U+FEFF is the invisible zero-width non-breaking space/ZWNBSP character). If the endian architecture of the decoder matches that of the encoder, the decoder detects the 0xFEFF value, but an opposite-endian decoder interprets the BOM as the noncharacter value U+FFFE reserved for this purpose. This incorrect result provides a hint to perform byte-swapping for the remaining values.

If the BOM is missing, RFC 2781 recommends that big-endian (BE) encoding be assumed. In practice, due to Windows using little-endian (LE) order by default, many applications assume little-endian encoding. It is also reliable to detect endianness by looking for null bytes, on the assumption that characters less than U+0100 are very common. If more even bytes (starting at 0) are null, then it is big-endian.

The standard also allows the byte order to be stated explicitly by specifying UTF-16BE or UTF-16LE as the encoding type. When the byte order is specified explicitly this way, a BOM is specifically not supposed to be prepended to the text, and a U+FEFF at the beginning should be handled as a ZWNBSP character. Most applications ignore a BOM in all cases despite this rule.

For Internet protocols, IANA has approved "UTF-16", "UTF-16BE", and "UTF-16LE" as the names for these encodings (the names are case insensitive). The aliases UTF_16 or UTF16 may be meaningful in some programming languages or software applications, but they are not standard names in Internet protocols.

Similar designations, UCS-2BE and UCS-2LE, are used to show versions of UCS-2.

Size

A "character" may use any number of Unicode code points. For instance an emoji flag character takes 8 bytes, since it is "constructed from a pair of Unicode scalar values" (and those values are outside the BMP and require 4 bytes each). UTF-16 in no way assists in "counting characters" or in "measuring the width of a string".

UTF-16 is often claimed to be more space-efficient than UTF-8 for East Asian languages, since it uses two bytes for characters that take 3 bytes in UTF-8. Since real text contains many spaces, numbers, punctuation, markup (for e.g. web pages), and control characters, which take only one byte in UTF-8, this is only true for artificially constructed dense blocks of text. A more serious claim can be made for Devanagari and Bengali, which use multi-letter words and all the letters take 3 bytes in UTF-8 and only 2 in UTF-16.

In addition the Chinese Unicode encoding standard GB 18030 always produces files the same size or smaller than UTF-16 for all languages, not just for Chinese (it does this by sacrificing self-synchronization).

Usage

UTF-16 is used for text in the OS API of all currently supported versions of Microsoft Windows (and including at least all since Windows CE/2000/XP/2003/Vista/7) including Windows 10. In Windows XP, no code point above U+FFFF is included in any font delivered with Windows for European languages. Older Windows NT systems (prior to Windows 2000) only support UCS-2. Files and network data tend to be a mix of UTF-16, UTF-8, and legacy byte encodings.

While there's been some UTF-8 support for even Windows XP, it was improved (in particular the ability to name a file using UTF-8) in Windows 10 insider build 17035 and the May 2019 update. As of May 2019, Microsoft recommends software use UTF-8, on Windows and Xbox, instead of other 8-bit encodings. It is unclear if they are recommending usage of UTF-8 over UTF-16, though they do state "UTF-16 is a unique burden that Windows places on code that targets multiple platforms."

The IBM i operating system designates CCSID (code page) 13488 for UCS-2 encoding and CCSID 1200 for UTF-16 encoding, though the system treats them both as UTF-16.

UTF-16 is used by the Qualcomm BREW operating systems; the .NET environments; and the Qt cross-platform graphical widget toolkit.

Symbian OS used in Nokia S60 handsets and Sony Ericsson UIQ handsets uses UCS-2. iPhone handsets use UTF-16 for Short Message Service instead of UCS-2 described in the 3GPP TS 23.038 (GSM) and IS-637 (CDMA) standards.

The Joliet file system, used in CD-ROM media, encodes file names using UCS-2BE (up to sixty-four Unicode characters per file name).

Python version 2.0 officially only used UCS-2 internally, but the UTF-8 decoder to "Unicode" produced correct UTF-16. There was also the ability to compile Python so that it used UTF-32 internally, this was sometimes done on Unix. Python 3.3 switched internal storage to use one of ISO-8859-1, UCS-2, or UTF-32 depending on the largest code point in the string. Python 3.12 drops some functionality (for CPython extensions) to make it easier to migrate to UTF-8 for all strings.

Java originally used UCS-2, and added UTF-16 supplementary character support in J2SE 5.0. Recently they have encouraged dumping support for any 8-bit encoding other than UTF-8 but internally UTF-16 is still used.

JavaScript may use UCS-2 or UTF-16. As of ES2015, string methods and regular expression flags have been added to the language that permit handling strings from an encoding-agnostic perspective.

UEFI uses UTF-16 to encode strings by default.

Swift, Apple's preferred application language, used UTF-16 to store strings until version 5 which switched to UTF-8.

Quite a few languages make the encoding part of the string object, and thus store and support a large set of encodings including UTF-16. Most consider UTF-16 and UCS-2 to be different encodings. Examples are the PHP language and MySQL.

A method to determine what encoding a system is using internally is to ask for the "length" of string containing a single non-BMP character. If the length is 2 then UTF-16 is being used. 4 indicates UTF-8. 3 or 6 may indicate CESU-8. 1 may indicate UTF-32, but more likely indicates the language decodes the string to code points before measuring the "length".

In many languages, quoted strings need a new syntax for quoting non-BMP characters, as the C-style "\uXXXX" syntax explicitly limits itself to 4 hex digits. The following examples illustrate the syntax for the non-BMP character U+1D11E 𝄞 MUSICAL SYMBOL G CLEF:

  • The most common (C++, C#, D, and several other languages) is an upper-case 'U' with 8 hex digits such as "\U0001D11E".
  • Java 7 regular expressions, ICU, and Perl, use "\x{1D11E}".
  • ECMAScript 2015 (JavaScript) uses "\u{1D11E}".
  • In many other cases (such as Java outside of regular expressions), the only way to get non-BMP characters is to enter the surrogate halves individually: "\uD834\uDD1E".

See also

Notes

  1. This number of code points is a consequence of the design of UTF-16
  2. UTF-32 is also incompatible with ASCII, but is not listed as a web-encoding.
  3. UTF-8 encoding produces byte values strictly less than 0xFE, so either byte in the BOM sequence also identifies the encoding as UTF-16 (assuming that UTF-32 is not expected).
  4. Use of U+FEFF as the character ZWNBSP instead of as a BOM has been deprecated in favor of U+2060 (WORD JOINER); see Byte Order Mark (BOM) FAQ at Unicode.org. But if an application interprets an initial BOM as a character, the ZWNBSP character is invisible, so the impact is minimal.
  5. RFC 2781 section 4.3 says that if there is no BOM, "the text SHOULD be interpreted as being big-endian." According to section 1.2, the meaning of the term "SHOULD" is governed by RFC 2119. In that document, section 3 says "... there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course".

References

  1. ^ "C.2 Encoding Forms in ISO/IEC 10646" (PDF). The Unicode Standard, version 6.0. Mountain View, CA: Unicode Consortium. February 2011. p. 573. ISBN 978-1-936213-01-6. the term UCS-2 should now be considered obsolete. It no longer refers to an encoding form in either 10646 or the Unicode Standard.
  2. ^ "FAQ: What is the difference between UCS-2 and UTF-16?". unicode.org. Archived from the original on 2003-08-18. Retrieved 2024-03-19. UCS-2 is obsolete terminology which refers to a Unicode implementation up to Unicode 1.1
  3. "What is UTF-16?". The Unicode Consortium. Unicode, Inc. Retrieved 7 January 2023. UTF-16 uses a single 16-bit code unit to encode over 60,000 of the most common characters in Unicode
  4. Lunde, Ken (2022-01-09). "2022 Top Ten List: Why Support Beyond-BMP Code Points?". Medium. Retrieved 2024-01-07. I first came up with the idea for this Top Ten List over 10 years ago, which was prompted by some environments that still supported only BMP code points. The idea, of course, was to motivate the developers of such environments to support code points beyond the BMP by providing an enumerated list of reasons to do so. And yes, there are still some environments that support only BMP code points, such as the VivaDesigner app.
  5. Chad Selph (2012-11-08). "Adventures in Unicode SMS". Twilio. Archived from the original on 2015-09-08. Retrieved 2015-08-28.
  6. "Should UTF-16 be considered harmful?". Software Engineering Stack Exchange. Retrieved 2024-11-20. File names editing in Window dialogs in broken (delete required 2 presses on backspace)
  7. "HTML Living Standard". w3.org. 2020-06-10. Archived from the original on 2020-09-08. Retrieved 2020-06-15. UTF-16 encodings are the only encodings that this specification needs to treat as not being ASCII-compatible encodings.
  8. "Encoding Standard". encoding.spec.whatwg.org. Retrieved 2023-04-22.
  9. "Usage Statistics of UTF-16 for Websites, September 2024". w3techs.com. Retrieved 2024-09-03.
  10. "Usage Statistics of UTF-8 for Websites, September 2024". w3techs.com. Retrieved 2024-09-03.
  11. "Encoding Standard". encoding.spec.whatwg.org. Retrieved 2018-10-22. The UTF-8 encoding is the most appropriate encoding for interchange of Unicode, the universal coded character set. Therefore for new protocols and formats, as well as existing formats deployed in new contexts, this specification requires (and defines) the UTF-8 encoding. The problems outlined here go away when exclusively using UTF-8, which is one of the many reasons that UTF-8 is now the mandatory encoding for all text things on the Web.
  12. "MySQL :: MySQL 5.7 Reference Manual :: 10.1.9.4 The ucs2 Character Set (UCS-2 Unicode Encoding)". dev.mysql.com.
  13. "What is UTF-16?". The Unicode Consortium. Unicode, Inc. Retrieved 29 March 2018.
  14. "Questions about encoding forms". Retrieved 2010-11-12.
  15. ISO/IEC 10646:2014 "Information technology – Universal Coded Character Set (UCS)" sections 9 and 10.
  16. The Unicode Standard version 7.0 (2014) section 2.5.
  17. "Unicode Character Encoding Stability Policies". unicode.org.
  18. Allen, Julie D.; Anderson, Deborah; Becker, Joe; Cook, Richard, eds. (2014). "3.8 Surrogates" (PDF). The Unicode Standard, Version 7.0—Core Specification. Mountain View: The Unicode Consortium. p. 118. Archived (PDF) from the original on 2022-10-09. Retrieved 3 November 2014.
  19. Yergeau, Francois; Hoffman, Paul (February 2000). "UTF-16, an encoding of ISO 10646". tools.ietf.org. Retrieved 2019-06-18.
  20. "Maximum Path Length Limitation". Microsoft. 2022-07-18. Retrieved 2022-10-10. the file system treats path and file names as an opaque sequence of WCHARs
  21. "It's not wrong that "🤦🏼‍♂️".length == 7". hsivonen.fi. Retrieved 2021-03-15.
  22. "Apple Developer Documentation". developer.apple.com. Retrieved 2021-03-15.
  23. Unicode (Windows). Retrieved 2011-03-08 "These functions use UTF-16 (wide character) encoding (…) used for native Unicode encoding on Windows operating systems."
  24. "Unicode". microsoft.com. Retrieved 2009-07-20.
  25. "Surrogates and Supplementary Characters". microsoft.com. Retrieved 2009-07-20.
  26. "Description of storing UTF-8 data in SQL Server". microsoft.com. 7 December 2005. Retrieved 2008-02-01.
  27. "[Updated] Patch for cmd.exe for windows xp for cp 65001 - Page 2 - DosTips.com". www.dostips.com. Retrieved 2021-06-17.
  28. "Use UTF-8 code pages in Windows apps". learn.microsoft.com. Retrieved 2020-06-06. As of Windows version 1903 (May 2019 update), you can use the ActiveCodePage property in the appxmanifest for packaged apps, or the fusion manifest for unpackaged apps, to force a process to use UTF-8 as the process code page. CP_ACP equates to CP_UTF8 only if running on Windows version 1903 (May 2019 update) or above and the ActiveCodePage property described above is set to UTF-8. Otherwise, it honors the legacy system code page. We recommend using CP_UTF8 explicitly.
  29. "UTF-8 support in the Microsoft Game Development Kit (GDK) - Microsoft Game Development Kit". learn.microsoft.com. Retrieved 2023-03-05. By operating in UTF-8, you can ensure maximum compatibility Windows operates natively in UTF-16 (or WCHAR), which requires code page conversions by using MultiByteToWideChar and WideCharToMultiByte. This is a unique burden that Windows places on code that targets multiple platforms. The Microsoft Game Development Kit (GDK) and Windows in general are moving forward to support UTF-8 to remove this unique burden of Windows on code targeting or interchanging with multiple platforms and the web. Also, this results in fewer internationalization issues in apps and games and reduces the test matrix that's required to get it right.
  30. "UCS-2 and its relationship to Unicode (UTF-16)". IBM. Retrieved 2019-04-26.
  31. Selph, Chad (2012-11-08). "Adventures in Unicode SMS". Twilio. Archived from the original on 2012-11-09. Retrieved 2015-08-28.
  32. "PEP 0393 – Flexible String Representation". Python.org. Retrieved 2015-05-29.
  33. "PEP 623 – Remove wstr from Unicode | peps.python.org". peps.python.org. Retrieved 2023-02-24.
  34. "JEP 400: UTF-8 by Default". openjdk.org. Retrieved 2023-03-12.
  35. "JavaScript's internal character encoding: UCS-2 or UTF-16? · Mathias Bynens".
  36. "UTF-8 String". Swift.org. 2019-03-20. Retrieved 2020-08-20.
  37. "PHP: Supported Character Encodings - Manual". php.net.
  38. "MySQL :: MySQL 8.0 Reference Manual :: 10.9.2 The utf8mb3 Character Set (3-Byte UTF-8 Unicode Encoding)". dev.mysql.com. Retrieved 2023-02-24.
  39. "ECMA-334: 9.4.1 Unicode escape sequences". en.csharp-online.net. Archived from the original on 2013-02-15.
  40. Lexical Structure: Unicode Escapes in "The Java Language Specification, Third Edition". Sun Microsystems, Inc. 2005. Retrieved 2019-10-11.

External links

Unicode
Unicode
Code points
Characters
Special purpose
Lists
Processing
Algorithms
Comparison of encodings
On pairs of
code points
Usage
Related standards
Related topics
Scripts and symbols in Unicode
Common and
inherited scripts
Modern scripts
Ancient and
historic scripts
Notational scripts
Symbols, emojis
Character encodings
Early telecommunications
ISO/IEC 8859
Bibliographic use
National standards
ISO/IEC 2022
Mac OS Code pages
("scripts")
DOS code pages
IBM AIX code pages
Windows code pages
EBCDIC code pages
DEC terminals (VTx)
Platform specific
Unicode / ISO/IEC 10646
TeX typesetting system
Miscellaneous code pages
Control character
Related topics
Character sets
Categories: