Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
As of Unicode 13.0.0 the characters U+FFFF, U+2FFFF, and U+1FFFF are unassgined code points, and as such have unpredictable behaviour.
The Unicode standard only suggests that unassigned code points be passed through, but implementations may be unable to operate on them effectively (as you see).
In glibc the unassigned code points have no guaranteed collation order (no assigned weights).
If you want an absolute code point ordering then there is a C.UTF-8 that we have in place for this purpose (full code point sorting), but it has some sorting defects that we are working on fixing.
May I ask what use you have for this data?
I disagree, codepoint U+FFFF is not assigned to any character but it does
have a well defined purpose.
http://www.unicode.org/reports/tr35/tr35-39/tr35-collation.html
----------------------------------------------------------------------
U+FFFF: This code point is tailored to have a primary weight
higher than all other characters. This allows the reliable
specification of a range, such as 'Sch' = X = 'Sch\uFFFF',
to include all strings starting with "sch" or equivalent.
U+FFFE: This code point produces a CE with minimal, unique
weights on primary and identical levels. For details see
the CLDR Collation Algorithm above.
----------------------------------------------------------------------
Again further down, in the same page.
----------------------------------------------------------------------
# SPECIAL MAX/MIN COLLATION ELEMENTS
FFFE; [02, 05, 05] # Special LOWEST primary, for merge/interleaving
FFFF; [EF FE, 05, 05] # Special HIGHEST primary, for ranges
The two tailored noncharacters have their own primary weights.
----------------------------------------------------------------------
Seems fairly clear what is intended here. This much should work
consistently regardless of locale.
>
> May I ask what use you have for this data?
U+FFFF is used to define a range search, like the standard says I can do.
In the case of U+2FFFF and U+1FFFF I put those in only for interest
sake, I don't need them. If the standard does not require any particular
collation order and there's no reason to break consistency, then they should
remain the same as previous major version RHEL-7. Alternatively perhaps
try to stay consistent with a reference library such as ICU.
(In reply to Telford Tendys from comment #2)
> I disagree, codepoint U+FFFF is not assigned to any character but it does
> have a well defined purpose.
Only for the Unicode Collation Algorithm.
If you wish to use the Unicode Collation Algorithm you must use a library that supports UCA like that available in libicu.
The collation in glibc is based on POSIX collation weights.
> U+FFFF is used to define a range search, like the standard says I can do.
Thank you for explaining what you were attempting to do.
You must consult your database documentation to determine if you can change your collation for your table to use ICU.
For exmaple, postgresql has a concept of collation provider:
https://www.postgresql.org/docs/11/collation.html
You can use "libc" for the defualt collation, or "icu" to use UCA.
In summary:
- In the libc collation (POSIX collation) there is no special purpose for U+FFFF.
- In the ICU implementation of UCA there is a special purpose for U+FFFF.
- Your database must be configured to use an implementation of UCA for collation to make use of U+FFFF.
Does this answer your question?
(In reply to Telford Tendys from comment #4)
> You have made yourself clear: this issue is unlikely to be fixed, and my
> only option is the ICU library.
>
> Thanks for the info, I guess that's all then.
Thanks for your understanding.
I'm going to mark this CLOSED/NOTABUG.
If you have any other questions please feel free to open a new bug.
Description of problem: Sort order is wrong, and has changed between RHEL/Centos 7 and RHEL/Centos 8. The effects all applications, e.g. SQL etc. Version-Release number of selected component (if applicable): glibc-2.28-151.el8.x86_64 glibc-common-2.28-151.el8.x86_64 glibc-devel-2.28-151.el8.x86_64 glibc-headers-2.28-151.el8.x86_64 glibc-langpack-en-2.28-151.el8.x86_64 How reproducible: Always Steps to Reproduce: 1. LC_ALL=C printf "\x78\x0a\x78\xf0\xaf\xbf\xbf\x0a\x78\xf0\x9f\xbf\xbf\x0a\x78\x2f\x0a\x78\xef\xbf\xbf\x0a\x78\x20\x0a" > /tmp/test1.utf8 2. LC_ALL=en_US.UTF-8 /bin/sort /tmp/test1.utf8 > /tmp/test2.utf8 3. perl -CSD -ne 'foreach my$c(unpack("U*")){printf("%02x\t",$c);};print"\n";' /tmp/test2.utf8 Actual results: 78 0a 78 ffff 0a 78 2ffff 0a 78 1ffff 0a 78 20 0a 78 2f 0a Expected results: 78 0a 78 20 0a 78 2f 0a 78 ffff 0a 78 2ffff 0a 78 1ffff 0a Additional info: Old Centos-7 system does produce the correct expected results.