Bug 657409

Hm, it dawned on me only now that maybe this is not a complaint about col.1.gz but /usr/bin/mandb is running col which is unhappy about something.  It is not obvious on quick glance, though, as /usr/bin/mandb is ELF 64-bit LSB executable.

Could you arrange for mandb to be run with the -d (or --debug) option the next time cron invokes it, and send me the output?

(In reply to comment #2)
> Could you arrange for mandb to be run with the -d (or --debug) option the next
> time cron invokes it, and send me the output?

It appears that mandb is too smart to fall for that. :-) After a check what it is doing is pretty obvious that it does not bother with pages already registered in its database.

OTOH dropping all index.db files from /var/cache/man and running it again forces mandb to check all available manpages.  With --debug this produces for me 35M of output (with both stdin and sterr redirected to a file).  Only there is no it this "Invalid" line from my report; so I have no idea where this came from.
Maybe a real offender was already replaced?

Still I collected 27 lines of "whatis parse for <something> failed" sort and two "ignoring bogus filename" (for manl/intro_blas1.man.gz and manl/ssyrk.txt_new.gz).  The later two already reported in April as bug #583182. These "failed" lines will show up also when not using --debug option and an output then is only 12K long.

Anybody wants to look at any of these two files?

For the record - these manpages make mandb unhappy:

Apache::TestServer.3pm.gz
checkXML.1.gz
CPAN::Debug.3pm.gz
CPAN::HandleConfig.3pm.gz
CPAN::Queue.3pm.gz
CPAN::Tarzip.3pm.gz
drpmsync.8.gz
faxformat.1.gz
kbuildsycoca4.8.gz
kcookiejar4.8.gz
kde4-config.1.gz
kded4.8.gz
kdeinit4.8.gz
kdeoptions.7.gz
kdesu.1.gz
kjs.1.gz
kjscmd.1.gz
kross.1.gz
lapack.l.gz
libpm.3.gz
meinproc4.8.gz
mpd.1.gz
nepomukserver.8.gz
nepomukservicestub.8.gz
PDA::Pilot.3pm.gz
qtoptions.7.gz
wireshark-filter.4.gz

At least the last one is bug #635878.

I'm seeing this, too.  Every day, cron.daily issues the complaint from col.  Running "mandb -c -d" from the command line does not show the complaint.

Doing some random checking in the language-specific man files turned up this:

% zcat /usr/share/man/bg/man1/deja-dup-preferences.1.gz | enca -L bulgarian
Universal transformation format 8 bits; UTF-8
  Doubly-encoded to UTF-8 from CP1251

Could doubly-encoded UTF-8 cause the error message?

I determined what was causing the "col: Invalid or incomplete multibyte or wide character" message on my system by making a change to the man-db.cron job. To locate the problem I used debug mode and redirected output of stdout and stderr. In man-db.cron the command change looks like this.

  #mandb $OPTS
  mandb -d > /root/stdout 2> /root/stderr

I then set up a temporary man-db.cron job to run using crontab. When the job finished I searched stderr for the col message and this is what I found.

...
Testing for existence: /usr/share/man/man8/groupadd.8
Testing for existence: /usr/share/man/man8/inadyn.8
inadyn(8) is not in the db.
Starting pipeline: gzip -dc /var/cache/man/cat8/inadyn.8.gz | /usr/libexec/man-d
b/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE | col -bx [input: {0, /var/cache/
man/cat8/inadyn.8.gz}, output: {-1, NULL}]
Started "gzip", pid 11976
Started "/usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE", pid 
11977
Started "col", pid 11978
Starting pipeline:  [input: {6, NULL}, output: {-1, NULL}]
trying encoding UTF-8 -> UTF-8//IGNORE
Waiting for pipeline:  [input: {6, NULL}, output: {-1, NULL}]
col: Invalid or incomplete multibyte or wide character
Waiting for pipeline: gzip -dc /var/cache/man/cat8/inadyn.8.gz | /usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE | col -bx [input: {0, /var/cache/man/cat8/inadyn.8.gz}, output: {-1, NULL}]
Active processes (3):
  "gzip" (11976) -> -1
  "/usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE" (11977) -> 0
  "col" (11978) -> -1
Active processes (2):
  "gzip" (11976) -> 0
  "col" (11978) -> -1
Active processes (1):
  "col" (11978) -> 256
record = 'inadyn-mt - a client for open DNS servers.'
base = 'inadyn'
name = 'inadyn-mt', id = E
name = 'inadyn', id = D
Testing for existence: /usr/share/man/man1/tftp.1
...

In my case it was the man file inadyn.8.gz. However, the inadyn-mt package was only on my system a short period of time and is no longer there. What remained behind was a copy of the man file inside the cache [/var/cache/man/cat8/inadyn.8.gz]. The col message stopped appearing after I removed that file from the cache.

I don't claim to know exactly what man-db is doing, but I do know that removing the offending file from the cache made the message go away. Perhaps mandb should clear the cache when a manpage is no longer in the db.

I was able to reproduce this bug by moving a man file from /usr/share/man that was in /var/cache/man. I first checked /var/cache/man/cat8 and found that mandb.8.gz was in the cache.  Next I moved mandb.8.gz from /usr/share/man/man8 to a temporary location. Then it was just a matter of running my modified cronjob and take a look at the results.

...
Testing for existence: /usr/share/man/man8/shutdown.8
Testing for existence: /usr/share/man/man8/mandb.8
mandb(8) is not in the db.
Starting pipeline: gzip -dc /var/cache/man/cat8/mandb.8.gz | /usr/libexec/man-db
/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE | col -bx [input: {0, /var/cache/m
an/cat8/mandb.8.gz}, output: {-1, NULL}]
Started "gzip", pid 9718
Started "/usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE", pid 
9719
Started "col", pid 9720
Starting pipeline:  [input: {6, NULL}, output: {-1, NULL}]
trying encoding UTF-8 -> UTF-8//IGNORE
Waiting for pipeline:  [input: {6, NULL}, output: {-1, NULL}]
col: Invalid or incomplete multibyte or wide character
Waiting for pipeline: gzip -dc /var/cache/man/cat8/mandb.8.gz | /usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE | col -bx [input: {0, /var/cache/man/cat8/mandb.8.gz}, output: {-1, NULL}]
Active processes (3):
  "gzip" (9718) -> -1
  "/usr/libexec/man-db/manconv -f UTF-8:ISO-8859-1 -t UTF-8//IGNORE" (9719) -> 0
  "col" (9720) -> -1
Active processes (2):
  "gzip" (9718) -> -1
  "col" (9720) -> 256
Active processes (1):
  "gzip" (9718) -> 0
record = 'mandb - create or update the manual page index caches'
base = 'mandb'
name = 'mandb', id = D
Testing for existence: /usr/share/man/man8/cron.8
Testing for existence: /usr/share/man/man8/usermod.8
...

You'll notice that the "col: Invalid or incomplete multibyte or wide character" message appeared. If I remove mandb.8.gz from the cache or restore it to /usr/share/man/man8 the message goes away.

This condition can occur if a package with a man file is installed and the man file viewed thereby putting it in the cache. After which the package is removed, but a man file from that package remains in the cache.

In my opinion this is not a problem with col itself, but how man-db is using col. I feel that this bug should be updated to make the component man-db and the Fedora version 14.

Is this related to the col man page using an endash instead of a dash?  i.e.:

.Dd May 2011 "  "
.Dt COL(1) "" "User Commands"
.Os util-linux
.Sh NAME
.Nm col
.Nd filter reverse line feeds from input

I've just seen this on Fedora 16.

The bug title is wrong - the error message is from an invocation of the col program by mandb, and has nothing to do with the col(1) manual page.

I haven't had a chance to pick through the logs yet, sorry; but, at least, William Makowski is correct that this should be reassigned to man-db.

(In reply to comment #8)
> The bug title is wrong - the error message is from an invocation of the col
> program by mandb, and has nothing to do with the col(1) manual page.

Yes.  See a follow up in a comment #1.

I also am seeing this with Fedora 16. It has been a consistent message that I have seen in Fedora 14 and 15 previously. So please update this to reflect Fedora 16 vice Fedora 14.

This bug has been reported elsewhere, possibly:

- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=446939
- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=555331
- https://bugs.archlinux.org/task/18722

William,

do you think you could try to reproduce this bug with this build and see whether it is fixed?

http://koji.fedoraproject.org/koji/buildinfo?buildID=311838

Thanks,

peter

I should have some time to run it through this weekend and will follow-up afterwards.

I downloaded the source from the build in comment #12 and performed a rebuild on a Fedora 16 box. It was also necessary to rebuild groff since groff-base was a dependency, looks like this split out as of Fedora 17.

Sorry, the "col: Invalid or incomplete multibyte or wide character" message continues to be a problem in man-db-2.6.1-1. I reproduced the bug using the method outlined in comment #6. The message also appears with the current version of man in Fedora 16, man-db-2.6.0.2-2.

William,

what's your locale? Please post output of:
$ locale


Thanks,

peter

Here is the output from locale.

[makowski@frodo ~]$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Thanks William. I have another task for you. Please, add these 2 lines to your modified man-db.cron file just before the mandb command and let me know, whether the error message still persists:

export LC_ALL="en_US.UTF-8"
export LANG="en_US.UTF-8"


Thanks,

peter

I had the same problem, and the following solved it for me:

rm -r /var/cache/man/*
/bin/bash /etc/cron.daily/man-db.cron

Peter,

You are onto something with setting LC_ALL and/or LANG within man-db.cron. I ran both with and without setting the variables using version man-db-2.6.0.2-3.fc16.i686. When they were set the error message did not occur.  When not set the "col: Invalid or incomplete multibyte or wide character" message came back.

Bill

Thank you William.

I've prepared testing build which could fix this. You can find it here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4176965

Shame is that I can't reproduce this bug anyhow. So, please, could you try to test it?

Thanks,

peter

My apologies for the delay. I do not see the src inside the testing build. Did I wait too long to pick them up?

Bill

Yeah, scratch builds last only for a week. I created another one here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=4215401

peter

Success! I performed a yum update using the man-db-2.6.2-0.1.fc18.i686.rpm on a Fedora 17 installation and it worked. The message is no longer coming up. Prior to the update I ran a test using man-db-2.6.0.2-6.fc17.i686 and the "col: Invalid or incomplete multibyte or wide character" message showed up.

Bill

fixed in:
man-db-2.6.2-1.fc18
http://koji.fedoraproject.org/koji/buildinfo?buildID=329909

man-db-2.6.0.2-7.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/man-db-2.6.0.2-7.fc17

man-db-2.6.0.2-4.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/man-db-2.6.0.2-4.fc16

Package man-db-2.6.0.2-7.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing man-db-2.6.0.2-7.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-10644/man-db-2.6.0.2-7.fc17
then log in and leave karma (feedback).

man-db-2.6.0.2-7.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

man-db-2.6.0.2-4.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.