5I would like to add a little comment. Most of Win32 "ASCII" functions receive locale strings in local encodings. For example std::ifstream can accept Hebrew file name if locale encoding is Hebrew one like 1255. Anything needed to support these encodings for windows is make MS add UTF-8 code page to the system. This would make the life much simpler. So all "ASCII" functions would be fully Unicode capable. – None – 2009-12-08T15:13:43.860

FWIW the AfxMessageBox(MfcUtils::Convert(s), _T("Error"), MB_OK) example should probably really have been a call to a wrapper of that function that accepts std::string(s). Also, the Assert(false) in the functions toward the end should be replaced with static assertions. – Assaf Lavie – 2009-12-09T03:38:35.220

39I can't agree. The advantages of utf16 over utf8 for many Asian languages completely dominate the points you make. It is naive to hope that the Japanese, Thai, Chinese, etc. are going to give up this encoding.

The problematic clashes between charsets are when the charsets mostly seem similar, except with differences.

I suggest standardising on: fixed 7bit: iso-irv-170; 8bit variable: utf8; 16bit variable: utf16; 32bit fixed: ucs4. – None – 2009-12-09T15:24:22.277

82@Charles: thanks for your input. True, some BMP characters are longer in UTF-8 than in UTF-16. But, let's face it: the problem is not in bytes that BMP Chinese characters take, but the software design complexity that arises.

If a Chinese programmer has to design for variable-length characters anyway, it seems like UTF-8 is still a small price to pay compared to other variables in the system.

He might use UTF-16 as a compression algorithm if space is so important, but even then it will be no match for LZ, and after LZ or other generic compression both take about the same size and entropy. – None – 2009-12-09T18:04:54.350

32What I basically say is that simplification offered by having One encoding that is also compatible with existing char* programs, and is also the most popular today for everything is unimaginable. It is almost like in good old "plaintext" days. Want to open a file with a name? No need to care what kind of unicode you are doing, etc etc.

I suggest we, developers, confine UTF-16 to very special cases of severe optimization where a tiny bit of performance is worth man-months of work. – None – 2009-12-09T18:08:35.030

5Well, if I had to choose between UTF-8 and UTF-16, I would definitely stick to UTF-8 as it has no BOM, ASCII-compliant and has the same encoding scheme for any plane. But I have to admit that UTF-16 is simpler and more efficient for most BMP characters. There's nothing worng with UTF-16 except the psychological aspects (mostly fixed-size isn't fixed size). Sure, one encoding would be better, but since both UTF-8 and UTF-16 are widely used, they have their advantages. – Malcolm – 2010-02-20T00:10:33.747

10@Malcolm: UTF-8, unfortunately, has a BOM too (0xEFBBBF). As silly as it looks (no byte order problem with single-byte encoding), this is true, and it is there for a different reason: to manifest this is a UTF stream. I have to disagree with you about BMP efficiency and UTF-16 popularity. It seems that majority of UTF-16 software do not support it properly (ex. all win32 API - which I am a fan of) and this is inherent, the easiest way to fix these seems to switch them to other encoding. The efficiency argument is only true for very narrow set of uses (I use hebrew, and even there it is not). – None – 2010-02-20T06:53:43.340

1Well, what I meant is that you don't have to worry about byte order. UTF-8 can have a BOM indeed (it is actually UTF-16 big endian BOM encoded in 3 bytes), though it's neither required, nor recommended according to the standard. As for the APIs I think the problem is that they were designed when surrogate pairs were either non-existent yet, or not really adopted. And when something gets patched up, it's always not as good as redesigning from the scratch. The only (painful) way is to drop any backwards compability and redesign the APIs. Should they switch to UTF-8 in the process, I don't know. – Malcolm – 2010-02-20T21:46:17.393

2@Malcolm, I think the natural way of this redesign is thru changing existing ANSI APIs. This way existing broken programs will unbreak (see my answer). This adds to the argument: UTF-16 must die. – None – 2010-03-07T08:14:09.533

3I'm sorry, I didn't really get the idea why transition to UTF-8 should be less painful. I also think that inconsistency in C++ makes it worse. Say, Java is very specific on the characters: char[] is no more than a char array, String is a string and Character is a character. Meanwhile, C++ is a mess with all the new stuff added to an existing language. To my mind, they should've abandoned any backwards compablity and design C++ in the way that doesn't allow to mix up structural programming and OOP or Unicode and other encodings. Not that I want to start a holy war, that's merely my opinion. – Malcolm – 2010-03-07T16:59:39.720

7UTF-8's disadvantage is NOT a small price to pay at all. Looking for any character is a O(n) operation, and other more complex operations can be far far worse than with UTF-16. Also UTF-8 is variable-length, just as UTF-16, so what's the point? UTF-8 was designed for storage and interoperability with ASCII. UTF-16 is the preferred way to store strings in memory, as anything outside the BMP is incredibly rare (you're wiring in Klingon?). With a little trick, storing characters outside of the BMP in a hash or map, UTF-16 can have constant processing time. – Mircea Chirea – 2010-03-17T14:18:57.620

2@iconiK: non-english BMP is also quite rare. Consider all program sources and markup languages. One should have very good reasons to use UTF-16. See what is going on in Linux world wrt unicode to measure the price of breaking changes. – None – 2010-03-17T16:02:26.353

17Linux has had a specific requirement when choosing to use UTF-8 internally: compatibility with Unix. Windows didn't need that, and thus when the developers implemented Unicode, they added UCS-2 versions of almost all functions handling text and made the multibyte ones simply convert to UCS-2 and call the other ones. THey later replaces UCS-2 with UTF-16. Linux on the other hand kept to 8-bit encodings and thus used UTF-8, as it's the proper choice in that case. – Mircea Chirea – 2010-03-17T17:56:19.983

4you may wish to read my answer again. Windows does not support UTF-16 properly to date. Also, the reason for choosing UCS-2 was different. Again, see my answer. For linux, I believe the main reason was compatibility not with unix but with existing code - for instance, if your ANSI app copies files, getting names from command arguments and calling system APIs, it will remain completely intact with UTF-8. Isn't that wonderful? – None – 2010-03-17T20:08:32.590

7@Pavel: The bug you linked to (Michael Kaplan's blog entry) has long been resolved by now. Michael said in the post already that it's fixed in Vista and I can't reproduce it on Windows 7 as well. While this doesn't fix legacy systems running on XP, saying that »there is still no proper support« is plain wrong. – Joey – 2010-04-02T13:37:45.597

@Johannes: [1] many thanks for the info. [2] IMO a programmer, today, should be able to write programs that support windows XP. It is still a popular one, and I don't know of a windows update that fixes it. – None – 2010-04-03T10:19:21.367

4Well, the program works just fine; it just has a little trouble dealing with astral planes, but that's an OS issue, not one with your program. It's like asking that current versions of Uniscribe are backported to old OSes that people on XP can enjoy a few scripts that would render improperly before. It's not something MS does. Besides, XP is almost a decade old by now and supporting it becomes a major burden in some cases (see for example the reasoning why Paint.NET will require Vista with its 4.0 release). Mainstream support for that OS has already ended, too; only security bugs are fixed now – Joey – 2010-04-03T11:22:57.620

2Still not convincing to use UTF-16 for in-memory presentation of strings on windows :) I wish Windows7 guys would extend their support of already existing #define of CP_UTF8 instead.. – None – 2010-04-03T17:50:32.070

3@Pavel Radzivilovsky: I fail to see how your code, using UTF-8 everywhere, will protect you from bugs in the Windows API? I mean: You're copying/converting strings for all calls to the WinAPI that use them, and still, if there is a bug in the GUI, or the filesystem, or whatever system handled by the OS, the bug remains. Now, perhaps your code has a specific UTF-8 handling functions (search for substrings, etc.), but then, you could have written them to handle UTF-16 instead, and avoid all this bloated code (unless you're writing cross-platform code... There, UTF-8 could be a sensible choice) – paercebal – 2010-09-04T12:20:17.040

34@Pavel Radzivilovsky: BTW, your writings about "I believe that all other encodings will die eventually. This involves that MS-Windows, Java, ICU, python stop using it as their favorite." and "In particular, I think adding wchar_t to C++ was a mistake, and so are the unicode additions to C++Ox." are either quite naive or very very arrogant. And this is coming from someone coding at home with a Linux and who is happy with the UTF-8 chars. To put it bluntly: It won't happen. – paercebal – 2010-09-04T12:28:19.953

2@paercebal: If majority of the code is API calls, this is a very simple code. Typically, majority of code dealing with strings is libraries that treat them as cookies, and they are optimized for. Hence, the bloating argument fails. As for the 'favorite utf16' for ICU and python, this is very questionable: these tools use UTF-16 internally, and changing it as a part of the evolution is the easiest. Can happen on any major release, coz it doesn't break the interfaces. – None – 2010-09-04T21:59:06.213

10In ICU we already see more and more UTF-8 interfaces and optimizations. However, UTF-16 works perfectly well, and makes complicated lookup efficient, more than with UTF-8. We will not see ICU drop UTF-16 internally. UTF-16 in memory, UTF-8 on the wire and on disk. All is good. – None – 2010-10-06T17:33:22.820

2@Steven, It looks like differentiating between wire and RAM is not a small thing as it may seem. BTW, comparison is cheaper with UTF8. I agree that ICU is certainly a major player on this market, and there's no need to "drop" support of anything. The simplification of application design and testing with UTF8 is exactly what will, in my humble opinion, drive UTf-16 to extinction, and the sooner the better. – None – 2010-10-08T14:50:51.823

3@Pavel Radzivilovsky I meant, drop UTF-16 as the internal processing format. Can you expand on 'not a small thing'? And, anyways, UTF-16/UTF-8/UTF-32 have a 1:1:1 mapping. I'm much more interested in seeing non-Unicode encodings die. As far as UTF-8 goes for simplification, you say "they can just pass strings as char"- right, and then they assume that the char is some ASCII-based 8-bit encoding. Plenty of errors creep in when toupper(), etc, is used on UTF-8. It's not wonderful, but it is helpful. – None – 2010-10-08T15:46:43.223

1@Steve First and foremost I agree about non-unicode. There's no argument about that. Practically, it already happened, they are already dead, in this exact sense: any non-unicode operation on a string is considered a bug just like any other software bug, or a 'text crime' in my company's slang. It is true that char* is misleading many into unicode bugs as well. Good luck with toupper() a UTF-8 string, or, say, with assuming that ICU toupper does not change the number of characters (as in german eszet converting to SS). After the standard has been established, there's no more reason to bug. – None – 2010-10-09T22:55:09.853

7@Steve, 2; and then we come to a more subtle thing, which is everything around human engineering and safety and designing proper way of work for a developer to do less and for the machine to do more. This is exactly where UTF-16 doesn't fit. Most applications do not reverse or even sort strings. Most often strings are treated as cookies, such as a file name here and there, concatenated here and there, embedded programming languages such as SQL and other really simple transformations. In this world, there's very little reason to have different format in RAM than on the wire. – None – 2010-10-09T23:00:12.347

1@Pavel: "...that widechar is going to be what UCS-4 now is." this is incorrect in general since widechar is not fixed to be 2 bytes in size, unless you restrict yourself to Windows. You should write "UCS-2" instead of "widechar". – None – 2010-12-10T10:16:32.097

2@ybungalobill Right; I should edit this. In fact, I will do this when wchar_t is standardized to hold one Unicode character. – None – 2010-12-11T22:51:35.290

10@Pavel: In fact your sentence is just wrong, because wchar_t is not meant to be UTF-16, it has absolutely no connection to UTF-16, and it is already UCS-4 on some compilers. wchar_t is a C++ type that is actually (quote from the standard) "a distinct type whose values can represent distinct codes for all members of the largest extended character set specified among the supported locales". So the only problem here is your system (windows) that doesn't have a UCS-4 locale. – None – 2010-12-12T09:36:43.680

9I'm mostly impressed with how this long rant completely fails in arguing its point, at least outside the narrow world of having to deal with UTF-16 in C and pointers. That might be considered dangerous, but that is if anything C's fault, not UTF-16. – Lennart Regebro – 2010-12-21T00:49:07.887

7Well, as I mentioned earlier, I didn't find this post very convincing either. This post goes into details of handling UTF-16 in certain APIs or languages. If the software doesn't handle the standard properly, that's a problem. But what's wrong with the encoding itself anyway? If some software implements only half of the standard, that's not the standard's problem. – Malcolm – 2010-12-29T16:56:31.787

4There are so many things that are wrong in the bullets that can't even be captures in a comment. But probably the most dangerous one is to store UTF-8 in std::string in a Windows environment. Problem is, everything in the Windows world assumes that char* are strings in the current system code page. Use one wrong API on that string, and you are assured of many hours of debugging. The other problem is the religious recommendation for UTF-8 no matter what. "there's a huge advantage to using the same encoding everywhere, and I see no sufficient reason to do otherwise" is pushed, no advantage given – None – 2011-05-28T08:54:40.067

3Wow, that's a really insightful comment. I'll start converting all my apps to UTF-8 right now. Thanks! – None – 2011-06-11T14:00:34.340

6@Mihai there is this advantage. you’ll start noticing it when you don’t do it and get cryptic runtime encoding exceptions nobody can possibly understand nor track back to it’s source. python 3 has made the jump, and guess what: the frequent encoding issues i had in python 2 magically disappeared completely. – flying sheep – 2011-08-04T06:34:31.527

1@flying sheep: And the reason you had problems in Python2 and not in Python3 is that Py3 is much stricter with distinguishing between Unicode and bytes. Apart from how you encode literal strings in your source file (and that can be changed to any other encoding if wanted without problems), you can't - or more exactly you SHOULDN'T - detect what encoding python is using internally. When communicating with the rest of the world you have to specify the encoding anyhow to avoid problems (otherwise you get a platform specific encoding). – Voo – 2011-08-15T15:21:40.060

1Yeah, I know. I love it this way: Input gets converted to Strings while reading it and the error happens there (if any), and not anywhere else in the code. Makes us guess less where the fuck it sneaked in. (Just like Scala’s Option keeps Scala programmers from encountering NullpointerExceptions: A implicit, error-prone process is replaced by a deliberate choice I make) – flying sheep – 2011-08-15T15:49:30.113

5Sorry but our programming languages should hide the encoding from us an implementation detail. We need datatypes that logically represent unicode characters without worrying about how they were stored on the harddrive. – None – 2011-08-18T15:52:24.430

1I'll agree to use a variable-length encoding for text when I am given an O(1) way of accessing a random character from a string. UCS2 and UCS4 have this nice ability, and are therefore surely better suited to internal uses than UTF-{8,16}? – holdenweb – 2011-08-19T05:35:45.977

13@holdenweb: Yyou cannot access a random character even in UCS-4, remember that character != codepoint. Moreover, there is absolutely no application where accessing nth character make sense. All the access to text is always sequential. – ybungalobill – 2011-08-21T12:19:41.340

1@Pavel: Why doesn't the answer contain an implementation for convert()? – Gili – 2012-03-22T02:32:06.640

11

Guys, thank you all for the feedback, both positive and negative! The discussion has inspired me and friends to publish a clear manifesto on the subject. Please enjoy, share and support: http://www.utf8everywhere.org/

1_T("") is not part ofthe standard, it is MS/Windows stuff. – ysdx – 2012-04-20T21:46:44.970

@PavelRadzivilovsky There is a memory leak in ::SetWindowTextW(Utils::convert(someStdString or "string litteral").c_str()) ..., isn't it? – kiewic – 2013-01-22T05:40:56.620

There is no leak - the memory is freed properly by the destructor of std::wstring – Pavel Radzivilovsky – 2013-01-22T12:32:50.373

1

Python has switched away from UCS-2 (which was only used for windows builds): http://python.org/dev/peps/pep-0393/

The very start of this answer is wrong The whole point of UTF16 is to encode 21-bit code points into a 16 bit wordspace – Michael Shaw – 2014-03-25T18:46:59.710

I strongly disagree with the "Don't use TCHAR" part, because TCHAR is actually the key for switching from "ANSI" and UTF-16 to UTF-8 with less pain. Create MessageBoxU8 as a different function from both MessageBoxA and MessageBoxW and do the same for all other string functions, and that way you can develop new programs that support UTF-8 without breaking the old ones that don't. – Medinoc – 2014-10-01T13:54:36.233

Dear Medinoc, please consider that one of the major points of the utf8everywhere.org manifesto is to say that you should never be switching between ANSI and any unicode encoding or support ANSI in the first place. There should be no non-unicode-aware programs written, compiled or tested. – Pavel Radzivilovsky – 2014-10-08T19:14:02.877

The footnote on Python should mention the https://docs.python.org/3/library/io.html#io.StringIO API, as that is the preferred way to manipulate large amounts of text that nevertheless still fit into memory. The fact that it exposes a file like seek/tell API without providing support for O(1) code point indexing lets it default to avoiding the memory cost of providing the latter, and also provide a better foundation for the kinds of cursor based algorithms needed to manipulate graphemes and characters rather than code points.

@ncoghlan, I would appreciate if you formulate the text to insert. Also, if you are into Python - it's high time to write to Python authors to change their internal string implementation to the better, UTF-8 way. – Pavel Radzivilovsky – 2015-06-01T15:32:22.633

1

As far as the text to insert goes: If you are manipulating text data in Python, and don't need O(1) code point indexing, then you should be using https://docs.python.org/3/library/io.html#io.StringIO as your data storage API, not str. https://docs.python.org/3/library/io.html#io.BytesIO is an alternative option for storing UTF-8 data directly in its encoded form.

19This. Very much this. UTF-16 can cause problems, but even using UTF-32 throughout can (and will) still give you issues. – bcat – 2010-12-24T00:48:17.007

11What is a character? You can define a code point as a character and get by pretty much just fine. If you mean a user-visible glyph, that’s something else. – tchrist – 2011-08-11T14:54:51.513

7@tchrist sure for allocating space that definition is fine, but for anything else? Not so much. If you handle a combining character as a sole character (ie for a delete or "take first N characters" operation) you'll get strange and wrong behavior. If a code point has only meaning when combined with at least another you can't handle it on its own in any sensible manner. – Voo – 2011-08-15T15:28:28.823

2bmp and astral is not the real issue. But allowing accented characters to have a separate diacritic is horrible. The guy who even thought of that idea deserved to have his head flushed in the toiletbowl – Pacerier – 2011-11-24T07:31:08.003

2This doesn't really answer the question. You have to deal with variable-length graphemes regardless of which encoding you use. However, variable-length code points are purely additional complexity that UTF-8 and UTF-16 have that UTF-32 doesn't. – dan04 – 2011-12-14T04:00:13.987

6@Pacerier, this is late to the party, but I have to comment on that. Some languages have very large sets of potential combinations of diacritics (c.f. Vietnamese, i.e. mệt đừ). Having combinations rather than one character per diacritic is very helpful. – syrion – 2012-04-20T21:23:24.380

21a small note on terminology: codepoints do correspond to unicode characters; what Daniel is talking about here are user-perceived characters, which correspond to unicode grapheme clusters – Christoph – 2012-04-21T11:58:39.763

6I agree with this answer. Internationalization is like software security : hard. There is all sort of edge case that you must account for, and the complexity is inherent to the domain. UTF-8 is great but its not a magic solution to make the pain go away. – Laurent Bourgault-Roy – 2012-12-19T18:44:47.107

2True, but do you really care about "counting"? When do you have to count codepoints at all? You may have to determine number or glyphs for rendering. But where is the use-case that requires counting codepoints? (we tend to think that this is important, because we so often need number or bytes of a buffer..). However, detecting & iterating codepoints is important for all kinds of text processing, but counting, no. – frunsi – 2013-01-29T22:16:44.753

Would upvote twice if I could. – Andrey Tarantsov – 2010-12-08T02:18:09.227

26Please add some examples where UTF-32 is common in the UNIX world! – maxschlepzig – 2011-01-13T13:39:50.163

48No, you do not want to use UTF-16 for data processing. It's a pain in the ass. It has all the disadvantages of UTF-8 but none of its advantages. Both UTF-8 and UTF-32 are clearly superior to the vicious hack previously known as Mrs UTF-16, whose maiden name was UCS-2. – tchrist – 2011-08-11T14:18:31.450

34I yesterday just found a bug in the Java core String class’s equalsIgnoreCase method (also others in the string class) that would never have been there had Java used either UTF-8 or UTF-32. There are millions of these sleeping bombshells in any code that uses UTF-16, and I am sick and tired of them. UTF-16 is a vicious pox that plagues our software with insidious bugs forever and ever. It is clearly harmful, and should be deprecated and banned. – tchrist – 2011-08-11T14:53:54.213

7@tchrist Wow so a non-surrogate aware function (because it was written when there were none and is sadly documented in such a way that makes it probably impossible to adapt - it specifies .toUpperCase(char)) will result in the wrong behavior? You're aware that a UTF-32 function with an outdated code point map wouldn't handle this any better? Also the whole Java API handles surrogates not especially well and the more intricate points about Unicode not at all - and with the later the used encoding wouldn't matter at all. – Voo – 2011-08-15T15:32:47.417

5There is absolutely no reason to use UTF-16 for data processing. – niXar – 2011-08-18T20:02:09.217

8-1: An unconditional .Substring(1) in .NET is a trivial example of something that break support for all of non-BMP Unicode. Everything that uses UTF-16 has this problem; it's too easy to treat it as a fixed-width encoding, and you see problems too rarely. That makes it an actively harmful encoding if you want to support Unicode. – romkyns – 2012-12-04T15:24:46.450

The problems mentioned in this thread are just sloppy programming, IMHO. Programmers should be able to deal with corner cases correctly, and programs should have unit tests to detect any such errors often. No one blames the concept of linked lists for the (alleged) difficulty of inserting/removing at the ends; the problem of surrogates in UTF-16 should be no different. – musiphil – 2014-06-18T19:35:50.163

@musiphil Carpenters should be able to handle corner cases too, but you don't see many of them building dodecahedrons or using 2x4x4x4s to do their framing. An intelligent programmer/carpenter will pick the tools and materials that will make their job easier unless specifications dictate otherwise. – technosaurus – 2014-07-15T21:28:15.323

16"Bugs related to not dealing with UTF-16 as a variable-width encoding are much more likely to go unnoticed than the equivalent bugs in UTF-8." This is the core of the issue, and hence, the correct answer. – Sean McMillan – 2011-08-19T13:01:09.850

3Precisely. If your UTF-8 handling is borked, it'll be immediately obvious. If your UTF-8 handling is borked, you'll only notice if you put in uncommon Han characters or math symbols. – Mechanical snail – 2012-08-01T07:26:08.557

1Very true, but on the other hand, what are unit tests for if you should depend on luck to find bugs on less frequent cases? – musiphil – 2014-06-18T21:09:55.667

@musiphil: so, when was the last time you created a unit test for non-BMP characters? – ninjalj – 2014-06-19T21:14:00.057

@ninjalj: I could be a text processing programmer, or a device driver programmer, or a graphics programmer; my answer to the specific question of yours shouldn't affect the truth/usefulness of the general statement that unit tests should be used to test corner cases. – musiphil – 2014-06-19T21:49:04.907

1To elaborate on my earlier statement: even with UTF-8, you cannot be assured that you have covered all cases after only seeing some working examples. Same with UTF-16: you need to test whether your code works both with non-surrogates and with surrogates. (Someone could even argue that UTF-8 has at least four major cases while UTF-16 has only two.) – musiphil – 2014-06-19T22:00:36.423

33-1: How about addressing some of Artyom's objections, rather than just patronising him? – None – 2009-06-26T16:12:32.690

8BTW: When I started writing this article I almost wanted to write "Does Joel on Softeare article of Unicode should be considered harmful" because there are many mistakes. For example: utf-8 encoding takes up to 4 characters and not 6. Also it does not distinguish between UCS-2 and UTF-16 that are really different -- and actually cause the problems I talk about. – None – 2009-06-26T16:12:41.443

1My point is that those character points are designed and implemented for specific tasks. The "bugs" you describe are no different than the "bugs" one would encounter if you attempted to give input outside the scope of any application. – None – 2009-06-26T16:21:48.823

2I agree with the last edit. The simplest example: we still use C and C++ though both languages use pointers and thus are not safe. – Malcolm – 2009-06-26T16:40:32.090

32Also, it should be noted that when Joel wrote that article, the UTF-8 standard WAS 6 bytes, not 4. RFC 3629 changed the standard to 4 bytes several months AFTER he wrote the article. Like most anything on the internet, it pays to read from more than one source, and to be aware of the age of your sources. The link wasn't intended to be the "end all be all", but rather a starting point. – None – 2009-06-26T16:42:24.727

1Actually, the problem is not with the standard. It is 100% ok. In fact, there are good implementations that work with utf-16: ICU, Java Swing etc. But, the problem is that there are too much basic bugs in processing of surragate pairs when working with utf-16, such, you should probably never pic utf-16 for internal encoding of new applications... Because there are lot of real life examples where utf-16 nature causes big troubles: even Stackoverlow can't deal with them – None – 2009-06-27T08:07:46.320

Not to try and flog a dead horse here, but if you shouldn't pick utf-16 as the reasonable standard, what should you pick? I'm interested in your perspective on what an acceptable alternative would be. For instance, a lot of my work involves working with ancient languages (greek, aramaic, hebrew, syriac, etc), and work a lot with these oddball unicode characters, so I'm constantly having to transition documents between utf-8, 16 and 32. – None – 2009-06-29T16:46:08.797

7I would pic: utf-8 or utf-32 that are: variable length encoding in almost all cases (including BMP) or fixed length encoding always. – None – 2009-07-12T06:50:35.830

2Artyom, SO doesn't NEED to use UTF-16, since UTF-8 is the de facto standard for storage and communication of text, while UTF-16 is the de facto standard for processing of text. I don't know of any web page using UTF-16, and it wouldn't be really bold to do so, especially since a really popular language has no Unicode support: PHP (and UTF-16 isn't really easy to deal with; UTF-8 is the standard encoding in most Linux installs, where PHP is commonly run). – Mircea Chirea – 2010-03-17T14:31:00.857

18@iconiK: Don’t be silly. UTF-16 is absolutely not the de facto standard for processing text. Show me a programming lanuage more suited to text processing that Perl, which has always (well, for more than a decade) used abstract characters with an underlying UTF-8 representation internally. Because of this, every Perl program automatically handles all Unicode without the user having to constantly monkey around with idiotic surrogates. The length of a string is its count in code points, not code units. Anything else is sheer stupidity putting the backwards into backwards compatibility. – tchrist – 2011-08-11T14:50:14.700

4@tchrist Great example! Because perl handles combined characters just as bad as any other well known programming language out there (its re module can handle them more sensibly, but its probably not the only one in that regard). abU+0063U+0327cde - a substring containing the first 3 characters really shouldn't return abc you know. So it doesn't handle "all of unicode perfectly" and the parts it does has to do with how it implemented its string library and not what encoding is used. – Voo – 2011-08-15T15:48:11.653

3

@Voo: I have no idea what you are talking about. Oh wait, maybe I do. The Unicode::GCString class has strings made of grapheme clusters instead of code points. perl -CS -MUnicode::GCString -le 'print Unicode::GCString::->new("ab\x{63}\x{327}cde")->substr(0, 3)' will dutifully print out abç (which is abc\x{327}). Isn’t that all you want? Piece of cake.

1@tchrist Nice, but that's not the default string implementation in perl is it? (at least not for my version!) Nobody argues that additional libraries can solve the problem correctly, but I don't see why they'd have a harder time if the codepoints were encoded with UTF-16, 8 or 32. – Voo – 2011-08-16T21:13:52.547

1What about foreach (var c in myString) in something like .NET? How often do you remember that this is UTF-16 you're dealing with? Let me guess: almost never. – romkyns – 2012-12-04T15:31:51.583

@tchrist: The fact that Perl programs automatically handles Unicode (up to some degree) has little to do with the fact that Perl uses UTF-8 internally; it would still hold if Perl used UTF-16 or UTF-32 internally, as there's nothing in UTF-16 or UTF-32 that hinders correct Unicode handling. – musiphil – 2014-06-18T21:05:04.823

12

In Java's case, if you look at their timeline (http://www.java.com/en/javahistory/timeline.jsp), you see that the primarily development of String happened while Unicode was 16 bits (it changed in 1996). They had to bolt on the ability to handle non BMP code points, thus the confusion.

10@Kathy: Not really an excuse for C#, though. Generally, I agree, that there should be a CodePoint type, holding a single code point (21 bits), a CodeUnit type, holding a single code unit (16 bits for UTF-16) and a Character type would ideally have to support a complete grapheme. But that makes it functionally equivalent to a String ... – Joey – 2010-04-02T13:43:10.843

1This answer is almost two years old, but I can't help but comment on it. "Having a type named 'char' which does not always represent a character is pretty confusing." And yet people use it all the time in C and the like to represent integer data that can be stored in a single byte. – JAB – 2011-06-06T15:53:55.303

And I've seen a lot of C code that doesn't handle character encoding correctly. – dan04 – 2011-08-18T23:06:31.437

1C# has a different excuse: it was designed for Windows, and Windows was built on UCS-2 (it's very annoying that even today Windows APIs cannot support UTF-8). Plus, I think Microsoft wanted Java compatibility (.NET 1.0 had a Java compatibility library, but they dropped Java support very quickly--I'm guessing this is due to Sun's lawsuit against MS?) – Qwertie – 2012-05-01T00:05:49.300

3UTF-16 is simpler for anything inside BMP, that's why it is used so widely. But I'm a fan of UTF-8 too, it also has no problems with byte order, which works to its advantage. – Malcolm – 2009-06-26T16:57:02.207

@Malcolm: UTF-16 also has no problems with byte order as it requires a BOM which specifies the order :-) – Joey – 2010-04-02T14:17:14.627

2Theoretically, yes. In practice there are such things as, say, UTF-16BE, which means UTF-16 in big endian without BOM. This is not some thing I made up, this is an actual encoding allowed in ID3v2.4 tags (ID3v2 tags suck, but are, unfortunately, widely used). And in such cases you have to define endianness externally, because the text itself doesn't contain BOM. UTF-8 is always written one way and it doesn't have such a problem. – Malcolm – 2010-04-02T15:33:53.830

23No, UTF-16 is not simpler. It is harder. It misleads and deceives you into thinking it is fixed width. All such code is broken and all the moreso because you don’t notice until it’s too late. CASE IN POINT: I just found yet another stupid UTF-16 bug in the Java core libraries yesterday, this time in String.equalsIgnoreCase, which was left in UCS-2 braindeath buggery, and so fails on 16/17 valid Unicode code points. How long has that code been around? No excuse for it to be buggy. UTF-16 leads to sheer stupidity and an accident waiting to happen. Run screaming from UTF-16. – tchrist – 2011-08-11T14:42:23.287

3

@tchrist One must be a very ignorant developer to not know that UTF-16 is not fixed length. If you start with Wikipedia, you will read the following at the very top: "It produces a variable-length result of either one or two 16-bit code units per code point". Unicode FAQ says the same: http://www.unicode.org/faq//utf_bom.html#utf16-1. I don't know, how UTF-16 can deceive anybody if it is written everywhere that it is variable length. As for the method, it was never designed for UTF-16 and shouldn't be considered Unicode, as simple as that.

@Malcolm: You write “One must be a very ignorant developer to not know that UTF-16 is not fixed length.” Well, welcome to the real world! At least 19/20 of them know about it at best extremely nebulously, and cannot even process a string by code points to save their lives. That is the reality. I know, because I’ve tested them on it. Until Java deprecates the whole char botchup you will always have crappy code full of BMP idiocies. All APIs need to be by int-32 code point, and the un-Unicode-Character char-16 versions deprecated into oblivion. Really they do. – tchrist – 2011-08-15T19:40:10.940

2@tchrist Do you have a source for your statistics? Though if good programmers a scarce, I think this is good, because we become more valuable. :) As for the Java APIs, char-based parts may eventually get deprecated, but this is not a guarantee that they won't be used. And they definitely won't be removed for compability reasons. – Malcolm – 2011-08-16T08:29:50.970

1@rmeador I would like to kill a widespread myth: UTF-8 is actually NOT compatible with any 8-bit based encoding. This is a fact that everyone in Europe should know. UTF-8 is backward compatible with US-ASCII, nothing more. – user877329 – 2013-07-13T09:35:36.277

1If a program uses UTF-16, shouldn't it be used "correctly"? – None – 2009-06-26T16:20:39.683

2Certainly. But that doesn't mean that if someone can use something incorrectly, we shouldn't use it at all, right? – Malcolm – 2009-06-26T16:22:17.437

21That's a rather blinkered, Anglo-centric view, Malcolm. Almost on a par with "ASCII is good enough for the USA - the rest of the world should fit in with us". – Jonathan Leffler – 2009-06-26T16:22:42.070

28Actually I'm from Russia and encounter cyrillics all the time (including my own programs), so I don't think that I have Anglo-centric view. :) Mentioning ASCII is not quite appropirate, because it's not Unicode and doesn't support specific characters. UTF-8, UTF-16, UTF-32 support the very same international character sets, they are just intended for use in their specific areas. And this is exactly my point: if you use mostly English, use UTF-8, if you use mostly cyrillics, use UTF-16, if you use ancient languages, use UTF-32. Quite simple. – Malcolm – 2009-06-26T16:36:02.027

6But you might not know in advance if your application need to handle characters outside BMP, if the application accepts data like names. For example some asian names might be written with characters outside of BMP. – None – 2009-06-26T21:28:59.583

2Not true, Asian scripts like Japanese, Chinese or Arabic belong to BMP also. BMP itself is actually very large and certainly large enough to include all the scripts used nowadays, it's not like it includes only European scripts or something. No, if you are really going to encounter non-BMP characters, you'll almost definitely know it. – Malcolm – 2009-06-27T09:52:34.470

1

@Malcolm: The issue is more complex than that. See eg. http://www.jbrowse.com/text/unij.html

And what did I write wrong? All characters of plane 2 contain only rare or historic symbols and all other characters fit into BMP and thus don't need surrogate pairs. – Malcolm – 2009-06-29T18:31:29.090

2@Malcolm: This issue is that some people apparently have names containing these rare symbols, even though they does not otherwise occur in regular language. – None – 2009-07-02T15:50:12.840

There is, but it's not really a problem specific for Unicode since standart encodings also don't include this characters. People use homophones and other ways to write such names, and that can be done in any encoding, including Unicode. Probably there are serious difficulties even with inputting rare symbols, so the situation doesn't happen all of a sudden and users won't be surprised to find out the program is refusing to accept them correctly if it does. – Malcolm – 2009-07-02T17:57:31.927

16"Not true, Asian scripts like Japanese, Chinese or Arabic belong to BMP also. BMP itself is actually very large and certainly large enough to include all the scripts used nowadays" This is all so wrong. BMP contains 0xFFFF characters (65536). Chinese alone has more than that. Chinese standards (GB 18030) has more than that. Unicode 5.1 already allocated more than 100,000 characters. – None – 2009-07-24T08:11:25.800

2It does, but characters outside BMP are not for everyday use, they can be used, for example, for old texts or to write names with rare hieroglyphs in them. And all characters that are commonly used fit into BMP. – Malcolm – 2009-08-12T17:57:18.630

12@Marcolm: "BMP itself is actually very large and certainly large enough to include all the scripts used nowadays"

Not true. At this point Unicode already allocated about 100K characters, way more than BMP can accomodate.

There are big chunks of Chinese characters outside BMP. And some of them are required by GB-18030 (mandatory Chinese standard). Other are required by (non-mandatory) Japanese and Korean standards. So if you try to sell anything in those markets, you need beyond BMP support. – None – 2009-09-25T21:41:25.870

4If BMP is that far from having enough capacity to write normally in Chinese, how do they manage to write in such encodings as GBK or GB 2312? It is clear that support of other planes would be useful, but nonetheless. – Malcolm – 2009-09-26T13:59:25.447

2All the currently used languages in the world fir in the BMP, in 64k code points. Anything outside of the BMP is not for current use of the language; it's for old characters, for old languages, for exotic characters, or even Klingon. If Chinese and/or Japanese and/or Korean need characters out of the BMP, how did they handle this before Unicode was widely adopted? Nearly all the encodings used in Asia were variable-length, using 8 or 16 bits per character. – Mircea Chirea – 2010-03-17T14:25:33.537

2You DO NOT want Klingon users to be angry at you ;-) – None – 2011-02-13T18:43:25.117

Why would they be angry? – Malcolm – 2011-02-13T19:42:32.250

8Anything that uses UTF-16 but can only handle narrow BMP characters is not actually using UTF-16. It is buggy and broken. The premise of the OP is sound: UTF-16 is harmful, because it leads naïve people into writing broken code. Either you can handle Unicode text, or you can’t. If you cannot, then you are picking a subset, which is just as stupid as ASCII-only text processing. – tchrist – 2011-08-11T14:46:46.403

2@tchrist Actually I think that most problems appear from the dated software which was designed for UCS-2. If you implement the standard today, you will be almost certainly aware that UTF-16 has a concept of surrogate pairs, since it is written about almost everywhere, starting with Wikipedia. And if you are an ignorant developer, you may not implement support for characters outside BMP in UTF-8 as well. Or even treat every text as if each byte represented only one character. – Malcolm – 2011-08-11T17:02:07.590

1@Malcolm: I have never seen a Go or Perl programmer get BMP screwups, and those languages both use UTF-8 internally. In contrast, in every language that uses UTF-16 I have seen people major screwups on BMP stuff everywhere I look. Where have you actually seen BMP screwups with UTF8-based programming languages? That seems utterly bizarre. Yes, you can do UTF-16 right, if you are genius smart and working on ICU. But for regular people, UTF-8 and UTF-32 are way less prone to error. – tchrist – 2011-08-11T20:30:08.073

@tchrist This is all too general. First of all, define what is a "BMP screwup" and where you look for them. Also, I've already said that there's a huge difference between something that was designed for UTF-16 and something that was designed for UCS-2 and then switched to UTF-16. And if we put aside API and language deficiencies, I don't reallly see what's so terribly difficult in handling surrogate pairs, especially in comparison with UTF-8. – Malcolm – 2011-08-12T09:07:40.680

1@Malcolm: The BMP screwup was iterating through a Java String a char at a time instead of code point at a time in the String class’s equalsIgnoreCase method. The code was never updated for UTF-16 so was stuck in UCS-2 brain damage, so does the wrong thing on anything outside the BMP. Plus it was using casemapping not casefolding, which is bound to get it into trouble. "", "", and "" are all casewise equal to each other, but the dumb UCS-2 Java method was too stupid to know that. ICU gets this right, and they use UTF-16 also, but not stupidly. – tchrist – 2011-08-12T09:17:20.727

2Yes, equalsIgnoreCase() compares chars, not codepoints, and it is stated in the docs. I certainly agree that if this method compared strings usign code points, it would be much simpler. But this isn't a problem of UTF-16 itself, it is a problem of a platform which was originally designed for UCS-2 - exactly what I'm talking about. – Malcolm – 2011-08-12T10:14:43.737

1@Malcom: I don’t believe that merely documenting something not to work on 16/17th of the Unicode space is acceptable. The correct solution is to make it do so. – tchrist – 2011-08-12T13:45:33.400

Consider it non-Unicode, that would clear the confusion. Historically this method is simply not meant to work reliably in conditions where I18N is required, it is not even locale-aware. Java has different facilities to do that. And I have to remind you that we're off the topic, still talking about subtleties of APIs, not about the UTF-16 itself. – Malcolm – 2011-08-12T16:08:59.863

2@Malcolm: Unicode is very much not just about i18n!! It is about dealing with text sanely. There are innumerable non-ASCII code points that go into purely English text, of which the most common are EN DASH and NO-BREAK SPACE, with the curly single- and double-quotes not far behind. Also, casing is not supposed to need to be locale aware; yes, I am very familiar with the Turkic I issue, but that is an exception case, so to speak. Unicode also has algorithms for line breaking of text, collation (sorting) of text, etc, that works way better than not using them even on pure English. Mythbusted! – tchrist – 2011-08-15T19:38:02.347

@tchrist I never said that Unicode and I18N are the same, though thanks for the info. – Malcolm – 2011-08-16T08:46:00.840

4-1 because you're defending using UTF-16 by denying the core purpose of Unicode: to unify all encoding. Having rare cases and basically ignoring them is a recipe for failure. – niXar – 2011-08-18T20:04:05.647

1@niXar If some software is ignoring surrogate pairs, then it isn't using UTF-16 and therefore is non-Unicode. You wouldn't call a program which works only with ASCII as supporting UTF-8, right? I think you misunderstood my point a bit, maybe I should clarify this in the answer. I say that there's nothing wrong with the UTF-16, but there's a lot of legacy software, which was designed for UCS-2, and therefore happens to be mostly compliant with UTF-16. But mostly compliant is not compliant, and this causes problems. I also doubt that if UTF-16 didn't exist at all, it would get fixed sooner. – Malcolm – 2011-08-18T21:17:51.967

1And I say there's plenty wrong with UTF-16, even ignoring UCS-2. – niXar – 2011-10-05T13:03:17.140

1@Malcolm: +1: a programmer's claiming to support UTF-16 but actually supporting no more than UCS-2 is not UTF-16's fault at all. – musiphil – 2014-06-18T21:20:18.440

6I didn't quite get your comment on UTF-8 and surrogate pairs. Surrogate pairs is only a concept that is meaningful in the UTF-16 encoding, right? Perhaps code that converts directly from UTF-16 encoding to UTF-8 encoding might get this wrong, and in that case, the problem is incorrectly reading the UTF-16, not writing the UTF-8. Is that right? – Craig McQueen – 2009-06-27T23:54:07.210

11What Jason's talking about is software that deliberately implements UTF-8 that way: create a surrogate pair, then UTF-8 encode each half separately. The correct name for that encoding is CESU-8, but Oracle (e.g.) misrepresents it as UTF-8. Java employs a similar scheme for object serialization, but it's clearly documented as "Modified UTF-8" and only for internal use. (Now, if we could just get people to READ that documentation and stop using DataInputStream#readUTF() and DataOutputStream#writeUTF() inappropriately...) – None – 2009-06-28T14:35:27.653

AFAIK, UTF-32 is still variable length encoding, and not equal to UCS4 which is specific range of code point. – Eonil – 2013-11-15T21:41:50.137

@Eonil, UTF-32 will only ever be distinguishable from UCS4 if we have a Unicode standard that features something like a UCS5 or larger. – JasonTrue – 2013-11-15T21:49:45.010

@JasonTrue Still, only the results are equal coincidently, not guaranteed by design. Same thing was happen in 32-bit memory addressing, Y2K, UTF16/UCS2. Or do we have any guarantee of that equality? If we have, I would gladly use that. But I don't want to write a possible breakable code. I am writing a character level code, and lack of a guaranteed way to transcode between UTF <-> code point is bugging me a lot. – Eonil – 2013-11-15T22:02:17.187

@JasonTrue I really want some guaranteed way badly. If you have any clue please help me to share that guarantee. – Eonil – 2013-11-15T22:02:59.420

I think you haven't read the standards document. That's the closest thing there is to a guarantee. It says: "UCS-4. UCS-4 stands for “Universal Character Set coded in 4 octets.” It is now treated sim- ply as a synonym for UTF-32, and is considered the canonical form for representation of characters in 10646." http://www.unicode.org/versions/Unicode6.2.0/appC.pdf

There's nothing complicated about transforming between UTF32 and UTF8, and it's only a tiny bit complicated to transform between UTF16 and UTF32. The code to do so reliably was included in the Unicode standard since at least 2.0 or so. – JasonTrue – 2013-11-15T22:21:17.090

@JasonTrue Thanks for that! I really needed it. I am sorry for not fully reading Unicode standard. Now I am on Unicode website, and see repeating mentions about that equality. – Eonil – 2013-11-15T23:33:31.663

9Constant time random access is possible with UTF-8 too, just use code units rather than code points.

Maybe you need real random code point access, but I've never seen a use case, and you're just as likely to want random grapheme cluster access instead. – None – 2010-08-06T07:32:19.577

Unicode guarantees there will be no codepoints above U+10FFFF. You are talking about UTF-32/UCS-4 (they are identical). If you are thinking about speed, 32->64 is not 16->32; int64 is not faster for 64-processors. – None – 2010-06-09T07:52:58.427

4@simon buchan, the U+10ffff max will go out the window when (not if) they run out of codepoints. That said, useing int32 on a p64 system for speed is probably safe, since i doubt they'll exceed U+ffffffff before you're forced to rewrite your code for 128 bit systems around 2050. (That is the point of "use the largest int that is convenient" as opposed to "largest available" (which would probably be int256 or bignums or something).) – David X – 2010-06-10T02:59:19.007

1@David: Unicode 5.2 encodes 107,361 codepoints. There are 867,169 unused codepoints. "when" is just silly. A Unicode codepoint is defined as a number from 0 to 0x10FFFF, a property which UTF-16 depends upon. (Also 2050 seems much to low an estimate for 128 bit systems when a 64-bit system can hold the entirety of the Internet in it's address space.) – None – 2010-06-11T06:07:58.823

@Simon, yes, I was thinking 2050 sounded a bit low for either ETA, my point was that yes, "when" is silly, but it will happen. My point in the original answer, however, was to use an array of ints of whatever size is needed for the largest codepoint you expect to handle. (And yes, I did forget that most p64 systems still use int32 as a primary integer type. I'm not sure why.) – David X – 2010-06-12T00:04:45.420

3@David: Your "when" was referring to running out of Unicode codepoints, not a 128-bit switch which, yes, will be in the next few centuries. Unlike memory, there is no exponential growth of characters, so the Unicode Consortium has specifically guaranteed they will never allocate a codepoint above U+10FFFF. This really is one of those situations when 21 bits is enough for anybody. – None – 2010-06-13T02:53:38.077

10@Simon Buchan: At least until first contact. :) – None – 2010-10-18T17:38:25.377

3Unicode used to guarantee that there would be no code points above U+FFFF too. – Shannon Severance – 2013-10-04T18:47:22.710

1Perhaps they haven't considered the exponential growth of smileys covering an ever-subtler range of emotions :mournfulwithachanceofjoy: – Robert Grant – 2014-08-28T07:11:53.867

7Note: UTF-16 does covers All Unicode as Unicode Consortium decided that 10FFFF is the TOP range of Unicode and defined UTF-8 maximal 4 bytes length and explicitly excluded range 0xD800-0xDFFF from valid code points range and this range is used for creation of surrogate pairs. So any valid Unicode text can be represented with each of one of these encodings. Also about growing to future. It doesn't seems that 1 Million code points would not be enough in any far future. – None – 2011-01-21T15:06:00.980

Exactly, all the encodings cover all the code points; and as for the lack of available codes, I don't see how this can be possible in forseeable future. Most supplementary planes are still unused, and even the used ones aren't full yet. So given the total sizes of the known writing systems left, it is very possible that most planes will never be used, unless they start to use code points for something different than writing systems. By the way, UTF-8 can theoretically include 6-byte sequences, so it can represent even more code points than UTF-32, but what's the point? – Malcolm – 2011-01-23T14:16:33.910

Malcolm: Not all encodings cover all codepoints. UCS-2 is if you will the fixed-size subset of UTF-16; it only covers the BMP. – Kerrek SB – 2011-06-09T11:34:12.150

7@Kerrek: Incorrect: UCS-2 is not a valid Unicode encoding. All UTF-* encodings by definition can represent any Unicode code point that is legal for interchange. UCS-2 can represent far fewer than that, plus a few more. Repeat: UCS-2 is not a valid Unicode encoding, any moreso than ASCII is. – tchrist – 2011-08-11T14:33:42.293

@_tchrist: You're right, UCS-2 isn't an encoding, it's a subset. In that sense, all encodings for Unicode must by definition be able to represent all Unicode codepoints. Fair point. – Kerrek SB – 2011-08-11T14:58:35.223

1"I do not understand advocating general use of Utf-8. It is variable length encoded (can not be accessed by index)" – Ian Boyd – 2011-08-11T15:35:35.013

9@Ian Boyd, the need to access a string’s individual character in a random access pattern is incredibly overstated. It is about as common as wanting to compute the diagonal of a matrix of characters, which is super rare. Strings are virtually always processed sequentially, and since accessing UTF-8 char N+1 given that you are at UTF-8 char N is O(1), there is no issue. There is surpassingly little need to make random access of strings. Whether you think it is worth the storage space to go to UTF-32 instead of UTF-8 is your own opinion, but for me, it is altogether a non-issue. – tchrist – 2011-08-11T20:38:36.550

2@tchrist, I will grant you strings are virtually always processed sequentially if you include reverse iteration as "sequential" and stretch that a little further comparison of the trailing end of a string to a known string. Two very common scenarios are truncating whitespace from the end of strings and checking the file extension at the end of a path. – Andy Dent – 2012-05-13T14:16:29.473

Thanks, very good link! I've added it to the issues list in the question. – None – 2010-12-21T06:28:50.667

8UTF-8 is a superset of ASCII, but UTF-16 is NOT a superset of UCS-2. Although almost a superset, a correct encoding of UCS-2 into UTF-8 results in the abomination known as CESU-8; UCS-2 doesn't have surrogates, just ordinary code points, so they must be translated as such.

The real advantage of UTF-16 is that it's easier to upgrade a UCS-2 codebase than a complete rewrite for UTF-8. Funny, huh? – None – 2010-08-06T07:28:27.000

1Sure, technically UTF-16 isn't a superset of UCS-2, but when were U+D800 to U+DFFF ever used for anything except UTF-16 surrogates? – dan04 – 2010-08-17T18:51:41.227

2Doesn't matter. Any processing other than blindly passing through the bytestream requires you to decode the surrogate pairs, which you can't do if you're treating it as UCS-2. – None – 2010-08-29T13:02:55.780

31-1 because UTF-8 is likely to be smaller or not significantly different. For certain Asian scripts UTF-8 is three bytes per glyph while UTF-16 is only two, but this is balanced by UTF-8 being only one byte for ASCII (which does often appear even within asian languages in product names, commands and such things). Further, in the said languages, a glyph conveys more information than a latin character so it is justified for it to take more space. – None – 2010-03-18T02:47:41.050

Thanks for the downvote, but I still don't get which part of the "best compromise between handling and space" you consider wrong. Note the word "compromise". Or maybe you don't believe that Win32, Java and .NET (also ICU, btw) use UTF-16 internally? – Nemanja Trifunovic – 2010-03-18T18:16:08.333

32I would not call combining the worst sides of both options a good compromise. – None – 2010-03-23T15:36:09.403

1It is the best of both worlds: it is pretty easy to handle, unlike UTF-8, and does not take nearly as much memory as UTF-32. – Nemanja Trifunovic – 2010-03-23T17:03:54.593

18It's not easier than UTF-8. It's variable-length too. – luiscubal – 2010-03-25T17:50:05.180

3

It is variable-length, but way easier than UTF-8. With UTF16, the only thing to look at is the surrogate pairs; a UTF-8 code point can be encoded as anyway between 1 and 4 bytes, plus you need to take care of things such as overlong sequences, etc. Look at this code to see how UTF-8 decoding looks like with C++: http://utfcpp.svn.sourceforge.net/viewvc/utfcpp/v2_0/source/utf8/core.h?revision=111&view=markup

36Leaving debates about the benefits of UTF-16 aside: What you cited is not the reason for Windows, Java or .NET using UTF-16. Windows and Java date back to a time where Unicode was a 16-bit encoding. UCS-2 was a reasonable choice back then. When Unicode became a 21-bit encoding migrating to UTF-16 was the best choice existing platforms had. That had nothing to do with ease of handling or space compromises. It's just a matter of legacy. – Joey – 2010-04-02T14:13:32.993

@Johannes: It is a matter of legacy in case of Win32 and Java, but not .NET and especially not Python 3. – Nemanja Trifunovic – 2010-04-02T15:37:43.347

10.NET inherits the Windows legacy here. – Joey – 2010-04-02T16:19:53.047

That's why I said "especially not Python 3", but it would have been perfectly feasible to implement even .NET strings as UTF-8. Of course, interop with Win32 is easier with UTF-16 strings. – Nemanja Trifunovic – 2010-04-02T17:51:04.543

Python3 and PHP6 are probably a case of "me too", and we all know how well that went with PHP6. – ninjalj – 2011-08-01T17:07:22.853

3

It does not in practice take a great deal more in UTF-8 than in UTF-16. See this case-study. Python3 is still unacceptably dodgy because you cannot rely on a wide build, so you never know how to count characters, or whether it takes "." or ".." to match one in a regex. Look at languages that have always used UTF-8, like Go and Perl, and you will see that they have none of the endless insanity of UTF-16. I just found another Java CORE UTF-16 bug yesteday.

5

@tchrist, you will be pleased to know that Python 3.3 fixes its internal representation of Unicode. See http://docs.python.org/py3k/whatsnew/3.3.html#pep-393

Agreed; thanks! We will update the document. The document still needs more development and adding information about databases. We are happy to receive contributions of wordings. – Pavel Radzivilovsky – 2014-02-14T12:41:08.997

@PavelRadzivilovsky _UNICODE is still there :( – cubuspl42 – 2014-04-18T12:19:05.947

thanks for reminding. cubus, Jelle, Would you like a user to our SVN? – Pavel Radzivilovsky – 2014-04-25T13:15:40.390

@Pavel Sure, would appreciate it! – Jelle Geerts – 2014-04-25T13:45:23.850

@JelleGeerts: I apologize for this delay. You could always contact us by our emails (linked from the manifesto) or Facebook. We are easy to find. Although I believe we fixed the issue you brought here (and I credited you there), the whole UTF-8 vs UTF-16 debates are still relevant. If you have more to contribute feel free to contact us through those private channels. – ybungalobill – 2015-06-24T19:03:05.387

This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post.

Unspecified endianess is supposed to include BOM as the first character, used for determining which way the string should be read. UCS-4 and UTF-32 indeed are the same nowadays, i.e. a numeric UCS value between 0 and 0x10FFFF stored in a 32 bit integer. – None – 2010-10-20T23:34:19.047

5@Tronic: Technically, this is not true. Although UCS-4 can store any 32-bit integer, UTF-32 is forbidden from storing the non-character code points that are illegal for interchange, such as 0xFFFF, 0xFFFE, and the all the surrogates. UTF is a transport encoding, not an internal one. – tchrist – 2011-08-11T14:30:15.463

Endianness issues are unavoidable as long as different processors continue to use different byte orders. However, it might have been nice if there were a "preferred" byte order for file storage of UTF-16. – Qwertie – 2012-05-01T00:16:14.777

Even though UTF-32 is fixed-width for code points, it is not fixed-width for characters. (Heard of something called "combining characters"?) So you can't go to the N'th character simply by indexing 4N into the byte array. – musiphil – 2014-06-19T05:05:35.873

3For those reading this comment, its worth noting that UCS-2 is not the same thing as UTF-16. Please look up the differences to understand. – mikebabcock – 2012-12-19T19:39:50.447

Knowing our luck, in a few years we'll find ourselves running out of space in UTF-16. Meh. – Donal Fellows – 2011-08-21T15:57:45.130

3The fundamental issue is that text is deceptively hard. No approach to representing that information in a digital way can be uncomplicated. It's the same reason that dates are hard, calendars are hard, time is hard, personal names are hard, postal addresses are hard: whenever digital machines intersect with human cultural constructs, complexity erupts. It’s a fact of life. Humans do not function on digital logic. – Aristotle Pagaltzis – 2012-05-07T01:16:48.433