Bug 1001775 - comm: file 1 is not in sorted order error generated during completion
comm: file 1 is not in sorted order error generated during completion
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: coreutils (Show other bugs)
20
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Ondrej Vasik
Fedora Extras Quality Assurance
: Reopened
: 1046735 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-27 13:32 EDT by Orion Poplawski
Modified: 2014-01-06 07:38 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-06 07:38:10 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Orion Poplawski 2013-08-27 13:32:33 EDT
Description of problem:

Trying to complete the "module avail" command I get:

comm: file 1 is not in sorted order

With set -x:

$ module load + local cur= prev=load cmds opts
+ COMPREPLY=()
+ cmds='add apropos avail clear display help          initadd initclear initlist initprepend initrm initswitch        keyword list load purge refresh rm show swap switch             unload unuse update use whatis'
+ opts='-c -f -h -i -l -s -t -u -v -H -V              --create --force  --help  --human   --icase             --long   --silent --terse --userlvl --verbose --version'
+ case "$prev" in
+ COMPREPLY=($(compgen -W "$(_module_not_yet_loaded)" -- "$cur"))
+++ _module_not_yet_loaded
+++ comm -23 /dev/fd/63 /dev/fd/62
++++ LANG=C
++++ sort
++++ LANG=C
++++ tr : '\n'
++++ sort
++++ _module_avail
++++ sed '
                /:$/d;
                /:ERROR:/d;
                s#^\(.*\)/\(.\+\)(default)#\1\n\1\/\2#;
                s#/(default)##g;
                s#/*$##g;'
++++ /usr/bin/modulecmd bash -t avail
comm: file 1 is not in sorted order
++ compgen -W 'compiler/intel/2011_sp1.11.339/i386
compiler/intel/2011_sp1.11.339/x86_64
compiler/intel/2011_sp1.12.361/i386
compiler/intel/2011_sp1.12.361/x86_64
compiler/intel/2011_sp1.13.367/i386
compiler/intel/2011_sp1.13.367/x86_64
compiler/intel/2013.0.079/i386
compiler/intel/2013.0.079/x86_64
compiler/intel/2013.1.117/i386
compiler/intel/2013.1.117/x86_64
compiler/intel/2013.2.146/i386
compiler/intel/2013.2.146/x86_64
compiler/intel/2013.3.163/i386
compiler/intel/2013.3.163/x86_64
compiler/intel/2013.5.192/i386
compiler/intel/2013.5.192/x86_64
compiler/pgi/12.5/x86_64
compiler/pgi/12.8/x86_64
compiler/pgi/13.2/x86_64
compiler/pgi/13.4/x86_64
compiler/pgi/13.5/x86_64
compiler/pgi/2012/x86_64
compiler/pgi/2013/x86_64
dot
module-git
module-info
modules
mpi/mpich-x86_64
mpi/openmpi-x86_64
mpich-x86_64
null
use.own' -- ''

Version-Release number of selected component (if applicable):
bash-completion-2.1-3.fc20.noarch

The environment-modules completion file (/usr/share/Modules/init/bash_completion) does run the entries through sort.
Comment 1 Ville Skyttä 2013-08-29 12:17:25 EDT
$ repoquery -f /usr/share/Modules/init/bash_completion
environment-modules-0:3.2.10-3.fc19.x86_64
Comment 2 Orion Poplawski 2013-08-30 01:08:08 EDT
Yeah, that's me :).  I would *love* to know what it is doing wrong.  It is running sort on the input files.  It seems to be a new error in F20/rawhide (works fine in F19).  Any help would be appreciated.  Thanks.
Comment 3 Orion Poplawski 2013-09-04 18:04:13 EDT
And now I don't see it.  Thanks rawhide.
Comment 4 Uwe Köcher 2013-12-17 05:12:13 EST
I encountered the same problem on the stable Fedora 20 release.

Adding the option --nocheck-order to the comm command in
/usr/share/Modules/bash_completion
such that _module_not_yet_loaded reads now as:

_module_not_yet_loaded() {
        comm --nocheck-order -23  <(_module_avail|sort)  <(tr : '\n' <<<${LOADEDMODULES}|sort)
}

resolved the problem.

Additional information:
$ repoquery -f /usr/share/Modules/init/bash_completion
gives for me:
environment-modules-0:3.2.10-6.fc20.x86_64
Comment 5 Orion Poplawski 2013-12-18 23:13:45 EST
@coreutils - Any idea why comm would complain that the output of the sort command is not in sorted order?
Comment 6 Pádraig Brady 2013-12-19 00:13:52 EST
If you could attach 2 sorted files that produce the error from comm that would help
Comment 7 Ondrej Vasik 2013-12-19 02:02:17 EST
Also the locales may be involved - try to use C/POSIX locales for the sort.
There is one downstream patch, changing sort between F19 and F20 - Ondrej Oprala did some optimizations and fixes - http://pkgs.fedoraproject.org/cgit/coreutils.git/commit/?id=0dcc5a0d5eb1554f4b5c0d2f2c8988b6b91e6553 - and there is already one known issue with this change (#1003544). Anyway, having these two files would be definitely helpful.
Comment 8 Uwe Köcher 2013-12-19 04:03:55 EST
Dear Ondrej and Pádraig,
I send you an email, where you can find the modulefiles.
The warning even occurs, when I try to use bash completion for
>$ module load <<completion>>

Note on comment #4:
the correct file is located at
/usr/share/Modules/init/bash_completion
sorry for that.
Comment 9 Orion Poplawski 2013-12-19 17:45:41 EST
Uwe - 

Could you do:

. /usr/share/Modules/init/bash_completion
_module_avail|sort > file1
tr : '\n' <<<${LOADEDMODULES}|sort > file2
comm -23 file1 file2

The comm command should output the not sorted order warning.

If so, please attach file1 and file2 to this report.  Also, what is the value of the LANG and LC_SORT environment variables?
Comment 10 Uwe Köcher 2013-12-20 03:46:00 EST
Orion,

sorry, your approach does not work,
file1 and file2 are attached below.
>$ echo $LANG
en_GB.UTF-8

>$ echo $LC_SORT
is empty.

It seems, that there are two kinds of sorting issues,
the first I guess is shown in the attached file: output,
the order between captial and non-captial letters:

NVRI.software/toolbox/FEniCS/2014.1
comm: file 1 is not in sorted order
compiler/mpi/mpich2/against-gcc-4.8.2/1.5

The second (see output2, where I removed all modulefiles
beginning with capital letters), is for special chars:

mpi/openmpi-x86_64
comm: file 1 is not in sorted order
mpich-x86_64

Best
  Uwe

file1:
NVRI.software/toolbox/DTM++.toolbox/2014.1
NVRI.software/toolbox/FEniCS/2014.1
compiler/mpi/mpich2/against-gcc-4.8.2/1.5
dot
gmsh/2.8.3
hdf5/mpich2/1.8.12
hdfview/2.9
jabref/2.9.2
maple/17
matlab/R2013a
matlab/R2013b
module-git
module-info
modules
mpi/mpich-x86_64
mpi/openmpi-x86_64
mpich-x86_64
null
openmp/init
paraview/4.0.1
toolbox.bak/mpich2
toolbox.bak/openmpi
toolbox/mpich2
use.own

file2:
compiler/mpi/mpich2/against-gcc-4.8.2/1.5
gmsh/2.8.3
hdfview/2.9
jabref/2.9.2
maple/17
matlab/R2013b
toolbox/mpich2
use.own

for output:
[koecheru@kitzbuehl ~]$ . /usr/share/Modules/init/bash_completion 
[koecheru@kitzbuehl ~]$ _module_avail|sort > file1
[koecheru@kitzbuehl ~]$ tr : '\n' <<<${LOADEDMODULES}|sort > file2
[koecheru@kitzbuehl ~]$ comm -23 file1 file2
NVRI.software/toolbox/DTM++.toolbox/2014.1
NVRI.software/toolbox/FEniCS/2014.1
comm: file 1 is not in sorted order
compiler/mpi/mpich2/against-gcc-4.8.2/1.5
dot
gmsh/2.8.3
hdf5/mpich2/1.8.12
hdfview/2.9
jabref/2.9.2
maple/17
matlab/R2013a
matlab/R2013b
module-git
module-info
modules
mpi/mpich-x86_64
mpi/openmpi-x86_64
mpich-x86_64
null
openmp/init
paraview/4.0.1
toolbox.bak/mpich2
toolbox.bak/openmpi


output2 (file1 does not contain NVRI*):
[koecheru@kitzbuehl ~]$ . /usr/share/Modules/init/bash_completion
[koecheru@kitzbuehl ~]$  _module_avail|sort > file1
[koecheru@kitzbuehl ~]$  tr : '\n' <<<${LOADEDMODULES}|sort > file2
[koecheru@kitzbuehl ~]$  comm -23 file1 file2
dot
hdf5/mpich2/1.8.12
matlab/R2013a
module-git
module-info
modules
mpi/mpich-x86_64
mpi/openmpi-x86_64
comm: file 1 is not in sorted order
mpich-x86_64
null
openmp/init
paraview/4.0.1
toolbox.bak/mpich2
toolbox.bak/openmpi
Comment 11 Orion Poplawski 2013-12-20 13:25:02 EST
I don't understand why sort would be putting the capitals first if LANG is not C.  Is LC_ALL set to anything?
Comment 12 Uwe Köcher 2013-12-20 14:12:41 EST
LC_ALL is empty.

Now I have installed F20 on 3 machines , all setted up from scratch (same packages/configurations as for F19), all with EN-gb locale.
The problem is present on all of this machines, on the other (F19) machines the output is correctly sorted.
Comment 13 Uwe Köcher 2013-12-20 15:34:14 EST
So, after reading manuals regarding sort, I've mentioned the first command from Comment #1.

>$ set -x
gives a lot of outputs on F20, one is:
++++ LC_ALL=C
on F19 only the last line is present (see below).
So something seems to set automatically LC_ALL to C.


The full output was:
++ __vte_prompt_command
+++ __vte_osc7
++++ __vte_urlencode /home/koecheru
++++ LC_ALL=C
++++ str=/home/****
++++ '[' -n /home/**** ']'
++++ safe=/home/***
++++ printf %s /home/****
++++ str=
++++ '[' -n '' ']'
++++ '[' -n '' ']'
+++ printf '\033]7;file://%s%s\a' *** /home/****
++ printf '\033]0;%s@%s:%s\007%s' **** *** '~' ''
Comment 14 Ondrej Vasik 2013-12-21 07:00:29 EST
Reproducible on my F20, comm(1) uses xmemcoll() gnulib function to compare the two lines in files and xmemcoll() really reports them as "unsorted". Have to check what differs - if it is xmemcoll() behaviour change in f20 or if sorting order changed for whatever reason.
Comment 15 Ondrej Vasik 2013-12-21 07:23:39 EST
In addition, F20 vanilla coreutils are not affected, it seems to be really caused by i18n changes done by O.Oprala in http://pkgs.fedoraproject.org/cgit/coreutils.git/commit/coreutils-i18n.patch?id=0dcc5a0d5eb1554f4b5c0d2f2c8988b6b91e6553 ... actually by the
+@@ -2689,14 +3311,6 @@ compare (struct line const *a, struct li
+ diff = - NONZERO (blen);
+ else if (blen == 0)
+ diff = 1;
+- else if (hard_LC_COLLATE)
+- {
+- /* Note xmemcoll0 is a performance enhancement as
+- it will not unconditionally write '\0' after the
+- passed in buffers, which was seen to give around
+- a 3% increase in performance for short lines. */
+- diff = xmemcoll0 (a->text, alen + 1, b->text, blen + 1);
+- }
+ else if (! (diff = memcmp (a->text, b->text, MIN (alen, blen))))
+ diff = alen < blen ? -1 : alen != blen;

When I revert this Ondrej's change, it sorts the file correctly, however it breaks some other multibyte tests - needs probably a bit different solution.
Comment 16 Ondrej Vasik 2013-12-26 15:18:27 EST
*** Bug 1046735 has been marked as a duplicate of this bug. ***
Comment 17 Ondrej Oprala 2014-01-06 07:38:10 EST
Thank you all for your contributions to solving this bugzilla. This issue should be fixed in the newest builds in f20/rawhide

Note You need to log in before you can comment on or make changes to this bug.