Bug 1001775
| Summary: | comm: file 1 is not in sorted order error generated during completion | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Orion Poplawski <orion> |
| Component: | coreutils | Assignee: | Ondrej Vasik <ovasik> |
| Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 20 | CC: | admiller, bugzilla, jsynacek, kdudka, khalasa, kzak, ooprala, orion, ovasik, pbrady, p, sheltren, twaugh |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-01-06 12:38:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
$ repoquery -f /usr/share/Modules/init/bash_completion environment-modules-0:3.2.10-3.fc19.x86_64 Yeah, that's me :). I would *love* to know what it is doing wrong. It is running sort on the input files. It seems to be a new error in F20/rawhide (works fine in F19). Any help would be appreciated. Thanks. And now I don't see it. Thanks rawhide. I encountered the same problem on the stable Fedora 20 release.
Adding the option --nocheck-order to the comm command in
/usr/share/Modules/bash_completion
such that _module_not_yet_loaded reads now as:
_module_not_yet_loaded() {
comm --nocheck-order -23 <(_module_avail|sort) <(tr : '\n' <<<${LOADEDMODULES}|sort)
}
resolved the problem.
Additional information:
$ repoquery -f /usr/share/Modules/init/bash_completion
gives for me:
environment-modules-0:3.2.10-6.fc20.x86_64
@coreutils - Any idea why comm would complain that the output of the sort command is not in sorted order? If you could attach 2 sorted files that produce the error from comm that would help Also the locales may be involved - try to use C/POSIX locales for the sort. There is one downstream patch, changing sort between F19 and F20 - Ondrej Oprala did some optimizations and fixes - http://pkgs.fedoraproject.org/cgit/coreutils.git/commit/?id=0dcc5a0d5eb1554f4b5c0d2f2c8988b6b91e6553 - and there is already one known issue with this change (#1003544). Anyway, having these two files would be definitely helpful. Dear Ondrej and Pádraig, I send you an email, where you can find the modulefiles. The warning even occurs, when I try to use bash completion for >$ module load <<completion>> Note on comment #4: the correct file is located at /usr/share/Modules/init/bash_completion sorry for that. Uwe -
Could you do:
. /usr/share/Modules/init/bash_completion
_module_avail|sort > file1
tr : '\n' <<<${LOADEDMODULES}|sort > file2
comm -23 file1 file2
The comm command should output the not sorted order warning.
If so, please attach file1 and file2 to this report. Also, what is the value of the LANG and LC_SORT environment variables?
Orion, sorry, your approach does not work, file1 and file2 are attached below. >$ echo $LANG en_GB.UTF-8 >$ echo $LC_SORT is empty. It seems, that there are two kinds of sorting issues, the first I guess is shown in the attached file: output, the order between captial and non-captial letters: NVRI.software/toolbox/FEniCS/2014.1 comm: file 1 is not in sorted order compiler/mpi/mpich2/against-gcc-4.8.2/1.5 The second (see output2, where I removed all modulefiles beginning with capital letters), is for special chars: mpi/openmpi-x86_64 comm: file 1 is not in sorted order mpich-x86_64 Best Uwe file1: NVRI.software/toolbox/DTM++.toolbox/2014.1 NVRI.software/toolbox/FEniCS/2014.1 compiler/mpi/mpich2/against-gcc-4.8.2/1.5 dot gmsh/2.8.3 hdf5/mpich2/1.8.12 hdfview/2.9 jabref/2.9.2 maple/17 matlab/R2013a matlab/R2013b module-git module-info modules mpi/mpich-x86_64 mpi/openmpi-x86_64 mpich-x86_64 null openmp/init paraview/4.0.1 toolbox.bak/mpich2 toolbox.bak/openmpi toolbox/mpich2 use.own file2: compiler/mpi/mpich2/against-gcc-4.8.2/1.5 gmsh/2.8.3 hdfview/2.9 jabref/2.9.2 maple/17 matlab/R2013b toolbox/mpich2 use.own for output: [koecheru@kitzbuehl ~]$ . /usr/share/Modules/init/bash_completion [koecheru@kitzbuehl ~]$ _module_avail|sort > file1 [koecheru@kitzbuehl ~]$ tr : '\n' <<<${LOADEDMODULES}|sort > file2 [koecheru@kitzbuehl ~]$ comm -23 file1 file2 NVRI.software/toolbox/DTM++.toolbox/2014.1 NVRI.software/toolbox/FEniCS/2014.1 comm: file 1 is not in sorted order compiler/mpi/mpich2/against-gcc-4.8.2/1.5 dot gmsh/2.8.3 hdf5/mpich2/1.8.12 hdfview/2.9 jabref/2.9.2 maple/17 matlab/R2013a matlab/R2013b module-git module-info modules mpi/mpich-x86_64 mpi/openmpi-x86_64 mpich-x86_64 null openmp/init paraview/4.0.1 toolbox.bak/mpich2 toolbox.bak/openmpi output2 (file1 does not contain NVRI*): [koecheru@kitzbuehl ~]$ . /usr/share/Modules/init/bash_completion [koecheru@kitzbuehl ~]$ _module_avail|sort > file1 [koecheru@kitzbuehl ~]$ tr : '\n' <<<${LOADEDMODULES}|sort > file2 [koecheru@kitzbuehl ~]$ comm -23 file1 file2 dot hdf5/mpich2/1.8.12 matlab/R2013a module-git module-info modules mpi/mpich-x86_64 mpi/openmpi-x86_64 comm: file 1 is not in sorted order mpich-x86_64 null openmp/init paraview/4.0.1 toolbox.bak/mpich2 toolbox.bak/openmpi I don't understand why sort would be putting the capitals first if LANG is not C. Is LC_ALL set to anything? LC_ALL is empty. Now I have installed F20 on 3 machines , all setted up from scratch (same packages/configurations as for F19), all with EN-gb locale. The problem is present on all of this machines, on the other (F19) machines the output is correctly sorted. So, after reading manuals regarding sort, I've mentioned the first command from Comment #1. >$ set -x gives a lot of outputs on F20, one is: ++++ LC_ALL=C on F19 only the last line is present (see below). So something seems to set automatically LC_ALL to C. The full output was: ++ __vte_prompt_command +++ __vte_osc7 ++++ __vte_urlencode /home/koecheru ++++ LC_ALL=C ++++ str=/home/**** ++++ '[' -n /home/**** ']' ++++ safe=/home/*** ++++ printf %s /home/**** ++++ str= ++++ '[' -n '' ']' ++++ '[' -n '' ']' +++ printf '\033]7;file://%s%s\a' *** /home/**** ++ printf '\033]0;%s@%s:%s\007%s' **** *** '~' '' Reproducible on my F20, comm(1) uses xmemcoll() gnulib function to compare the two lines in files and xmemcoll() really reports them as "unsorted". Have to check what differs - if it is xmemcoll() behaviour change in f20 or if sorting order changed for whatever reason. In addition, F20 vanilla coreutils are not affected, it seems to be really caused by i18n changes done by O.Oprala in http://pkgs.fedoraproject.org/cgit/coreutils.git/commit/coreutils-i18n.patch?id=0dcc5a0d5eb1554f4b5c0d2f2c8988b6b91e6553 ... actually by the +@@ -2689,14 +3311,6 @@ compare (struct line const *a, struct li + diff = - NONZERO (blen); + else if (blen == 0) + diff = 1; +- else if (hard_LC_COLLATE) +- { +- /* Note xmemcoll0 is a performance enhancement as +- it will not unconditionally write '\0' after the +- passed in buffers, which was seen to give around +- a 3% increase in performance for short lines. */ +- diff = xmemcoll0 (a->text, alen + 1, b->text, blen + 1); +- } + else if (! (diff = memcmp (a->text, b->text, MIN (alen, blen)))) + diff = alen < blen ? -1 : alen != blen; When I revert this Ondrej's change, it sorts the file correctly, however it breaks some other multibyte tests - needs probably a bit different solution. *** Bug 1046735 has been marked as a duplicate of this bug. *** Thank you all for your contributions to solving this bugzilla. This issue should be fixed in the newest builds in f20/rawhide |
Description of problem: Trying to complete the "module avail" command I get: comm: file 1 is not in sorted order With set -x: $ module load + local cur= prev=load cmds opts + COMPREPLY=() + cmds='add apropos avail clear display help initadd initclear initlist initprepend initrm initswitch keyword list load purge refresh rm show swap switch unload unuse update use whatis' + opts='-c -f -h -i -l -s -t -u -v -H -V --create --force --help --human --icase --long --silent --terse --userlvl --verbose --version' + case "$prev" in + COMPREPLY=($(compgen -W "$(_module_not_yet_loaded)" -- "$cur")) +++ _module_not_yet_loaded +++ comm -23 /dev/fd/63 /dev/fd/62 ++++ LANG=C ++++ sort ++++ LANG=C ++++ tr : '\n' ++++ sort ++++ _module_avail ++++ sed ' /:$/d; /:ERROR:/d; s#^\(.*\)/\(.\+\)(default)#\1\n\1\/\2#; s#/(default)##g; s#/*$##g;' ++++ /usr/bin/modulecmd bash -t avail comm: file 1 is not in sorted order ++ compgen -W 'compiler/intel/2011_sp1.11.339/i386 compiler/intel/2011_sp1.11.339/x86_64 compiler/intel/2011_sp1.12.361/i386 compiler/intel/2011_sp1.12.361/x86_64 compiler/intel/2011_sp1.13.367/i386 compiler/intel/2011_sp1.13.367/x86_64 compiler/intel/2013.0.079/i386 compiler/intel/2013.0.079/x86_64 compiler/intel/2013.1.117/i386 compiler/intel/2013.1.117/x86_64 compiler/intel/2013.2.146/i386 compiler/intel/2013.2.146/x86_64 compiler/intel/2013.3.163/i386 compiler/intel/2013.3.163/x86_64 compiler/intel/2013.5.192/i386 compiler/intel/2013.5.192/x86_64 compiler/pgi/12.5/x86_64 compiler/pgi/12.8/x86_64 compiler/pgi/13.2/x86_64 compiler/pgi/13.4/x86_64 compiler/pgi/13.5/x86_64 compiler/pgi/2012/x86_64 compiler/pgi/2013/x86_64 dot module-git module-info modules mpi/mpich-x86_64 mpi/openmpi-x86_64 mpich-x86_64 null use.own' -- '' Version-Release number of selected component (if applicable): bash-completion-2.1-3.fc20.noarch The environment-modules completion file (/usr/share/Modules/init/bash_completion) does run the entries through sort.