As of Unicode 13.0.0 the characters U+FFFF, U+2FFFF, and U+1FFFF are unassgined code points, and as such have unpredictable behaviour.

The Unicode standard only suggests that unassigned code points be passed through, but implementations may be unable to operate on them effectively (as you see).

In glibc the unassigned code points have no guaranteed collation order (no assigned weights).

If you want an absolute code point ordering then there is a C.UTF-8 that we have in place for this purpose (full code point sorting), but it has some sorting defects that we are working on fixing.

May I ask what use you have for this data?

I disagree, codepoint U+FFFF is not assigned to any character but it does
have a well defined purpose.

http://www.unicode.org/reports/tr35/tr35-39/tr35-collation.html

----------------------------------------------------------------------
    U+FFFF: This code point is tailored to have a primary weight
    higher than all other characters. This allows the reliable
    specification of a range, such as 'Sch' = X = 'Sch\uFFFF',
    to include all strings starting with "sch" or equivalent.

    U+FFFE: This code point produces a CE with minimal, unique
    weights on primary and identical levels. For details see
    the CLDR Collation Algorithm above.
----------------------------------------------------------------------


Again further down, in the same page.

----------------------------------------------------------------------
    # SPECIAL MAX/MIN COLLATION ELEMENTS
    FFFE; [02, 05, 05]     # Special LOWEST primary, for merge/interleaving
    FFFF; [EF FE, 05, 05]  # Special HIGHEST primary, for ranges

    The two tailored noncharacters have their own primary weights.
----------------------------------------------------------------------


Seems fairly clear what is intended here. This much should work
consistently regardless of locale.


>
> May I ask what use you have for this data?

U+FFFF is used to define a range search, like the standard says I can do.

In the case of U+2FFFF and U+1FFFF I put those in only for interest
sake, I don't need them. If the standard does not require any particular
collation order and there's no reason to break consistency, then they should
remain the same as previous major version RHEL-7. Alternatively perhaps
try to stay consistent with a reference library such as ICU.

(In reply to Telford Tendys from comment #2)
> I disagree, codepoint U+FFFF is not assigned to any character but it does
> have a well defined purpose.

Only for the Unicode Collation Algorithm.

If you wish to use the Unicode Collation Algorithm you must use a library that supports UCA like that available in libicu.

The collation in glibc is based on POSIX collation weights.

> U+FFFF is used to define a range search, like the standard says I can do.

Thank you for explaining what you were attempting to do.

You must consult your database documentation to determine if you can change your collation for your table to use ICU.

For exmaple, postgresql has a concept of collation provider:
https://www.postgresql.org/docs/11/collation.html

You can use "libc" for the defualt collation, or "icu" to use UCA.

In summary:
- In the libc collation (POSIX collation) there is no special purpose for U+FFFF.
- In the ICU implementation of UCA there is a special purpose for U+FFFF.
- Your database must be configured to use an implementation of UCA for collation to make use of U+FFFF.

Does this answer your question?

You have made yourself clear: this issue is unlikely to be fixed, and my only option is the ICU library.

Thanks for the info, I guess that's all then.

(In reply to Telford Tendys from comment #4)
> You have made yourself clear: this issue is unlikely to be fixed, and my
> only option is the ICU library.
> 
> Thanks for the info, I guess that's all then.

Thanks for your understanding.

I'm going to mark this CLOSED/NOTABUG.

If you have any other questions please feel free to open a new bug.