Bug 1975045
| Summary: | UTF-8 codepoint FFFF sorts low, should sort high. | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Telford Tendys <bugzilla> |
| Component: | glibc | Assignee: | glibc team <glibc-bugzilla> |
| Status: | CLOSED NOTABUG | QA Contact: | qe-baseos-tools-bugs |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | CentOS Stream | CC: | ashankar, bstinson, codonell, dj, fweimer, jwboyer, mnewsome, pfrankli, sipoyare |
| Target Milestone: | beta | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-07-02 15:48:37 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
As of Unicode 13.0.0 the characters U+FFFF, U+2FFFF, and U+1FFFF are unassgined code points, and as such have unpredictable behaviour. The Unicode standard only suggests that unassigned code points be passed through, but implementations may be unable to operate on them effectively (as you see). In glibc the unassigned code points have no guaranteed collation order (no assigned weights). If you want an absolute code point ordering then there is a C.UTF-8 that we have in place for this purpose (full code point sorting), but it has some sorting defects that we are working on fixing. May I ask what use you have for this data? I disagree, codepoint U+FFFF is not assigned to any character but it does have a well defined purpose. http://www.unicode.org/reports/tr35/tr35-39/tr35-collation.html ---------------------------------------------------------------------- U+FFFF: This code point is tailored to have a primary weight higher than all other characters. This allows the reliable specification of a range, such as 'Sch' = X = 'Sch\uFFFF', to include all strings starting with "sch" or equivalent. U+FFFE: This code point produces a CE with minimal, unique weights on primary and identical levels. For details see the CLDR Collation Algorithm above. ---------------------------------------------------------------------- Again further down, in the same page. ---------------------------------------------------------------------- # SPECIAL MAX/MIN COLLATION ELEMENTS FFFE; [02, 05, 05] # Special LOWEST primary, for merge/interleaving FFFF; [EF FE, 05, 05] # Special HIGHEST primary, for ranges The two tailored noncharacters have their own primary weights. ---------------------------------------------------------------------- Seems fairly clear what is intended here. This much should work consistently regardless of locale. > > May I ask what use you have for this data? U+FFFF is used to define a range search, like the standard says I can do. In the case of U+2FFFF and U+1FFFF I put those in only for interest sake, I don't need them. If the standard does not require any particular collation order and there's no reason to break consistency, then they should remain the same as previous major version RHEL-7. Alternatively perhaps try to stay consistent with a reference library such as ICU. (In reply to Telford Tendys from comment #2) > I disagree, codepoint U+FFFF is not assigned to any character but it does > have a well defined purpose. Only for the Unicode Collation Algorithm. If you wish to use the Unicode Collation Algorithm you must use a library that supports UCA like that available in libicu. The collation in glibc is based on POSIX collation weights. > U+FFFF is used to define a range search, like the standard says I can do. Thank you for explaining what you were attempting to do. You must consult your database documentation to determine if you can change your collation for your table to use ICU. For exmaple, postgresql has a concept of collation provider: https://www.postgresql.org/docs/11/collation.html You can use "libc" for the defualt collation, or "icu" to use UCA. In summary: - In the libc collation (POSIX collation) there is no special purpose for U+FFFF. - In the ICU implementation of UCA there is a special purpose for U+FFFF. - Your database must be configured to use an implementation of UCA for collation to make use of U+FFFF. Does this answer your question? You have made yourself clear: this issue is unlikely to be fixed, and my only option is the ICU library. Thanks for the info, I guess that's all then. (In reply to Telford Tendys from comment #4) > You have made yourself clear: this issue is unlikely to be fixed, and my > only option is the ICU library. > > Thanks for the info, I guess that's all then. Thanks for your understanding. I'm going to mark this CLOSED/NOTABUG. If you have any other questions please feel free to open a new bug. |
Description of problem: Sort order is wrong, and has changed between RHEL/Centos 7 and RHEL/Centos 8. The effects all applications, e.g. SQL etc. Version-Release number of selected component (if applicable): glibc-2.28-151.el8.x86_64 glibc-common-2.28-151.el8.x86_64 glibc-devel-2.28-151.el8.x86_64 glibc-headers-2.28-151.el8.x86_64 glibc-langpack-en-2.28-151.el8.x86_64 How reproducible: Always Steps to Reproduce: 1. LC_ALL=C printf "\x78\x0a\x78\xf0\xaf\xbf\xbf\x0a\x78\xf0\x9f\xbf\xbf\x0a\x78\x2f\x0a\x78\xef\xbf\xbf\x0a\x78\x20\x0a" > /tmp/test1.utf8 2. LC_ALL=en_US.UTF-8 /bin/sort /tmp/test1.utf8 > /tmp/test2.utf8 3. perl -CSD -ne 'foreach my$c(unpack("U*")){printf("%02x\t",$c);};print"\n";' /tmp/test2.utf8 Actual results: 78 0a 78 ffff 0a 78 2ffff 0a 78 1ffff 0a 78 20 0a 78 2f 0a Expected results: 78 0a 78 20 0a 78 2f 0a 78 ffff 0a 78 2ffff 0a 78 1ffff 0a Additional info: Old Centos-7 system does produce the correct expected results.