Bug 657409 - util-linux-ng supplied /usr/share/man/man1/col.1.gz is not liked by man-db
util-linux-ng supplied /usr/share/man/man1/col.1.gz is not liked by man-db
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: man-db (Show other bugs)
rawhide
Unspecified Unspecified
low Severity medium
: ---
: ---
Assigned To: Peter Schiffer
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-11-25 15:18 EST by Michal Jaegermann
Modified: 2012-07-23 16:23 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-07-20 22:54:09 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Michal Jaegermann 2010-11-25 15:18:20 EST
Description of problem:

As a side-effect of an update of man-db the following showed up in a cron output:

/etc/cron.daily/man-db.cron:

col: Invalid or incomplete multibyte or wide character

OTOH 'man col' seems to display just fine and a cursory look at a content of /usr/share/man/man1/col.1.gz does not make anything jump on me as "obvious".
What has really a problem here?

Version-Release number of selected component (if applicable):
util-linux-ng-2.18-5.fc15.x86_64
man-db-2.5.9-1.fc15.x86_64
Comment 1 Michal Jaegermann 2010-11-25 15:25:22 EST
Hm, it dawned on me only now that maybe this is not a complaint about col.1.gz but /usr/bin/mandb is running col which is unhappy about something.  It is not obvious on quick glance, though, as /usr/bin/mandb is ELF 64-bit LSB executable.
Comment 2 Colin Watson 2010-12-02 08:45:30 EST
Could you arrange for mandb to be run with the -d (or --debug) option the next time cron invokes it, and send me the output?
Comment 3 Michal Jaegermann 2010-12-02 21:04:16 EST
(In reply to comment #2)
> Could you arrange for mandb to be run with the -d (or --debug) option the next
> time cron invokes it, and send me the output?

It appears that mandb is too smart to fall for that. :-) After a check what it is doing is pretty obvious that it does not bother with pages already registered in its database.

OTOH dropping all index.db files from /var/cache/man and running it again forces mandb to check all available manpages.  With --debug this produces for me 35M of output (with both stdin and sterr redirected to a file).  Only there is no it this "Invalid" line from my report; so I have no idea where this came from.
Maybe a real offender was already replaced?

Still I collected 27 lines of "whatis parse for <something> failed" sort and two "ignoring bogus filename" (for manl/intro_blas1.man.gz and manl/ssyrk.txt_new.gz).  The later two already reported in April as bug #583182. These "failed" lines will show up also when not using --debug option and an output then is only 12K long.

Anybody wants to look at any of these two files?

For the record - these manpages make mandb unhappy:

Apache::TestServer.3pm.gz
checkXML.1.gz
CPAN::Debug.3pm.gz
CPAN::HandleConfig.3pm.gz
CPAN::Queue.3pm.gz
CPAN::Tarzip.3pm.gz
drpmsync.8.gz
faxformat.1.gz
kbuildsycoca4.8.gz
kcookiejar4.8.gz
kde4-config.1.gz
kded4.8.gz
kdeinit4.8.gz
kdeoptions.7.gz
kdesu.1.gz
kjs.1.gz
kjscmd.1.gz
kross.1.gz
lapack.l.gz
libpm.3.gz
meinproc4.8.gz
mpd.1.gz
nepomukserver.8.gz
nepomukservicestub.8.gz
PDA::Pilot.3pm.gz
qtoptions.7.gz
wireshark-filter.4.gz

At least the last one is bug #635878.
Comment 4 Jerry James 2010-12-14 23:51:26 EST
I'm seeing this, too.  Every day, cron.daily issues the complaint from col.  Running "mandb -c -d" from the command line does not show the complaint.

Doing some random checking in the language-specific man files turned up this:

% zcat /usr/share/man/bg/man1/deja-dup-preferences.1.gz | enca -L bulgarian
Universal transformation format 8 bits; UTF-8
  Doubly-encoded to UTF-8 from CP1251

Could doubly-encoded UTF-8 cause the error message?
Comment 5 William Makowski 2011-03-17 10:38:28 EDT
I determined what was causing the "col: Invalid or incomplete multibyte or wide character" message on my system by making a change to the man-db.cron job. To locate the problem I used debug mode and redirected output of stdout and stderr. In man-db.cron the command change looks like this.

  #mandb $OPTS
  mandb -d > /root/stdout 2> /root/stderr

I then set up a temporary man-db.cron job to run using crontab. When the job finished I searched stderr for the col message and this is what I found.

...
Testing for existence: /usr/share/man/man8/groupadd.8
Testing for existence: /usr/share/man/man8/inadyn.8
inadyn(8) is not in the db.
Starting pipeline: gzip -dc /var/cache/man/cat8/inadyn.8.gz | /usr/libexec/man-d
b/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE | col -bx [input: {0, /var/cache/
man/cat8/inadyn.8.gz}, output: {-1, NULL}]
Started "gzip", pid 11976
Started "/usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE", pid 
11977
Started "col", pid 11978
Starting pipeline:  [input: {6, NULL}, output: {-1, NULL}]
trying encoding UTF-8 -> UTF-8//IGNORE
Waiting for pipeline:  [input: {6, NULL}, output: {-1, NULL}]
col: Invalid or incomplete multibyte or wide character
Waiting for pipeline: gzip -dc /var/cache/man/cat8/inadyn.8.gz | /usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE | col -bx [input: {0, /var/cache/man/cat8/inadyn.8.gz}, output: {-1, NULL}]
Active processes (3):
  "gzip" (11976) -> -1
  "/usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE" (11977) -> 0
  "col" (11978) -> -1
Active processes (2):
  "gzip" (11976) -> 0
  "col" (11978) -> -1
Active processes (1):
  "col" (11978) -> 256
record = 'inadyn-mt - a client for open DNS servers.'
base = 'inadyn'
name = 'inadyn-mt', id = E
name = 'inadyn', id = D
Testing for existence: /usr/share/man/man1/tftp.1
...

In my case it was the man file inadyn.8.gz. However, the inadyn-mt package was only on my system a short period of time and is no longer there. What remained behind was a copy of the man file inside the cache [/var/cache/man/cat8/inadyn.8.gz]. The col message stopped appearing after I removed that file from the cache.

I don't claim to know exactly what man-db is doing, but I do know that removing the offending file from the cache made the message go away. Perhaps mandb should clear the cache when a manpage is no longer in the db.
Comment 6 William Makowski 2011-04-04 11:39:30 EDT
I was able to reproduce this bug by moving a man file from /usr/share/man that was in /var/cache/man. I first checked /var/cache/man/cat8 and found that mandb.8.gz was in the cache.  Next I moved mandb.8.gz from /usr/share/man/man8 to a temporary location. Then it was just a matter of running my modified cronjob and take a look at the results.

...
Testing for existence: /usr/share/man/man8/shutdown.8
Testing for existence: /usr/share/man/man8/mandb.8
mandb(8) is not in the db.
Starting pipeline: gzip -dc /var/cache/man/cat8/mandb.8.gz | /usr/libexec/man-db
/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE | col -bx [input: {0, /var/cache/m
an/cat8/mandb.8.gz}, output: {-1, NULL}]
Started "gzip", pid 9718
Started "/usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE", pid 
9719
Started "col", pid 9720
Starting pipeline:  [input: {6, NULL}, output: {-1, NULL}]
trying encoding UTF-8 -> UTF-8//IGNORE
Waiting for pipeline:  [input: {6, NULL}, output: {-1, NULL}]
col: Invalid or incomplete multibyte or wide character
Waiting for pipeline: gzip -dc /var/cache/man/cat8/mandb.8.gz | /usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE | col -bx [input: {0, /var/cache/man/cat8/mandb.8.gz}, output: {-1, NULL}]
Active processes (3):
  "gzip" (9718) -> -1
  "/usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE" (9719) -> 0
  "col" (9720) -> -1
Active processes (2):
  "gzip" (9718) -> -1
  "col" (9720) -> 256
Active processes (1):
  "gzip" (9718) -> 0
record = 'mandb - create or update the manual page index caches'
base = 'mandb'
name = 'mandb', id = D
Testing for existence: /usr/share/man/man8/cron.8
Testing for existence: /usr/share/man/man8/usermod.8
...

You'll notice that the "col: Invalid or incomplete multibyte or wide character" message appeared. If I remove mandb.8.gz from the cache or restore it to /usr/share/man/man8 the message goes away.

This condition can occur if a package with a man file is installed and the man file viewed thereby putting it in the cache. After which the package is removed, but a man file from that package remains in the cache.

In my opinion this is not a problem with col itself, but how man-db is using col. I feel that this bug should be updated to make the component man-db and the Fedora version 14.
Comment 7 Tim Waugh 2011-12-01 06:43:14 EST
Is this related to the col man page using an endash instead of a dash?  i.e.:

.Dd May 2011 "  "
.Dt COL(1) "" "User Commands"
.Os util-linux
.Sh NAME
.Nm col
.Nd filter reverse line feeds from input

I've just seen this on Fedora 16.
Comment 8 Colin Watson 2011-12-01 07:09:53 EST
The bug title is wrong - the error message is from an invocation of the col program by mandb, and has nothing to do with the col(1) manual page.

I haven't had a chance to pick through the logs yet, sorry; but, at least, William Makowski is correct that this should be reassigned to man-db.
Comment 9 Michal Jaegermann 2011-12-01 11:52:28 EST
(In reply to comment #8)
> The bug title is wrong - the error message is from an invocation of the col
> program by mandb, and has nothing to do with the col(1) manual page.

Yes.  See a follow up in a comment #1.
Comment 10 Stephen Haffly 2011-12-09 10:50:21 EST
I also am seeing this with Fedora 16. It has been a consistent message that I have seen in Fedora 14 and 15 previously. So please update this to reflect Fedora 16 vice Fedora 14.
Comment 12 Peter Schiffer 2012-04-05 10:59:03 EDT
William,

do you think you could try to reproduce this bug with this build and see whether it is fixed?

http://koji.fedoraproject.org/koji/buildinfo?buildID=311838

Thanks,

peter
Comment 13 William Makowski 2012-04-05 23:11:53 EDT
I should have some time to run it through this weekend and will follow-up afterwards.
Comment 14 William Makowski 2012-04-09 13:46:56 EDT
I downloaded the source from the build in comment #12 and performed a rebuild on a Fedora 16 box. It was also necessary to rebuild groff since groff-base was a dependency, looks like this split out as of Fedora 17.

Sorry, the "col: Invalid or incomplete multibyte or wide character" message continues to be a problem in man-db-2.6.1-1. I reproduced the bug using the method outlined in comment #6. The message also appears with the current version of man in Fedora 16, man-db-2.6.0.2-2.
Comment 15 Peter Schiffer 2012-05-24 10:05:56 EDT
William,

what's your locale? Please post output of:
$ locale


Thanks,

peter
Comment 16 William Makowski 2012-05-24 11:23:07 EDT
Here is the output from locale.

[makowski@frodo ~]$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Comment 17 Peter Schiffer 2012-05-24 16:44:52 EDT
Thanks William. I have another task for you. Please, add these 2 lines to your modified man-db.cron file just before the mandb command and let me know, whether the error message still persists:

export LC_ALL="en_US.UTF-8"
export LANG="en_US.UTF-8"


Thanks,

peter
Comment 18 Peter Backes 2012-05-30 22:40:29 EDT
I had the same problem, and the following solved it for me:

rm -r /var/cache/man/*
/bin/bash /etc/cron.daily/man-db.cron
Comment 19 William Makowski 2012-06-07 15:27:47 EDT
Peter,

You are onto something with setting LC_ALL and/or LANG within man-db.cron. I ran both with and without setting the variables using version man-db-2.6.0.2-3.fc16.i686. When they were set the error message did not occur.  When not set the "col: Invalid or incomplete multibyte or wide character" message came back.

Bill
Comment 20 Peter Schiffer 2012-06-19 09:59:23 EDT
Thank you William.

I've prepared testing build which could fix this. You can find it here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4176965

Shame is that I can't reproduce this bug anyhow. So, please, could you try to test it?

Thanks,

peter
Comment 21 William Makowski 2012-07-02 15:51:19 EDT
My apologies for the delay. I do not see the src inside the testing build. Did I wait too long to pick them up?

Bill
Comment 22 Peter Schiffer 2012-07-03 08:49:35 EDT
Yeah, scratch builds last only for a week. I created another one here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4215401

peter
Comment 23 William Makowski 2012-07-04 09:08:44 EDT
Success! I performed a yum update using the man-db-2.6.2-0.1.fc18.i686.rpm on a Fedora 17 installation and it worked. The message is no longer coming up. Prior to the update I ran a test using man-db-2.6.0.2-6.fc17.i686 and the "col: Invalid or incomplete multibyte or wide character" message showed up.

Bill
Comment 24 Peter Schiffer 2012-07-10 14:34:04 EDT
fixed in:
man-db-2.6.2-1.fc18
http://koji.fedoraproject.org/koji/buildinfo?buildID=329909
Comment 25 Fedora Update System 2012-07-12 12:26:43 EDT
man-db-2.6.0.2-7.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/man-db-2.6.0.2-7.fc17
Comment 26 Fedora Update System 2012-07-12 12:52:07 EDT
man-db-2.6.0.2-4.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/man-db-2.6.0.2-4.fc16
Comment 27 Fedora Update System 2012-07-14 17:58:06 EDT
Package man-db-2.6.0.2-7.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing man-db-2.6.0.2-7.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-10644/man-db-2.6.0.2-7.fc17
then log in and leave karma (feedback).
Comment 28 Fedora Update System 2012-07-20 22:54:09 EDT
man-db-2.6.0.2-7.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 29 Fedora Update System 2012-07-23 16:23:13 EDT
man-db-2.6.0.2-4.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.