From Bugzilla Helper: User-Agent: Mozilla/5.0 Galeon/1.2.6 (X11; Linux i686; U;) Gecko/20020916 Description of problem: When using the DateManip perl module, stderr is filled with "Malformed UTF-8 character..." messages. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. use Date::Manip in a perl script 2. 3. Actual Results: Stuff like this appears on stdout: (but Date::Manip appears to actually work fine) Malformed UTF-8 character (byte 0xfe) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5902. Malformed UTF-8 character (unexpected non-continuation byte 0x6d, immediately after start byte 0xee) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5902. Malformed UTF-8 character (unexpected non-continuation byte 0x74, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5902. Malformed UTF-8 character (1 byte, need 3, after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5902. Malformed UTF-8 character (1 byte, need 3, after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5903. Malformed UTF-8 character (unexpected non-continuation byte 0x6d, immediately after start byte 0xee) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5906. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5926. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5927. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5929. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5929. Malformed UTF-8 character (unexpected non-continuation byte 0x73, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5930. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5930. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5930. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5930. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5931. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5931. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5931. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5931. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5932. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5932. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5932. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5932. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5933. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5933. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5933. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5933. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5933. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5933. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5934. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5934. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5934. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5934. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5935. Malformed UTF-8 character (unexpected non-continuation byte 0x6e, immediately after start byte 0xee) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5943. Malformed UTF-8 character (unexpected non-continuation byte 0x69, immediately after start byte 0xee) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5943. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5943. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5943. Malformed UTF-8 character (1 byte, need 3, after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5944. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5945. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5946. Malformed UTF-8 character (unexpected non-continuation byte 0x73, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5946. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5946. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5947. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5947. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5947. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5947. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5947. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5947. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5948. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5948. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5948. Malformed UTF-8 character (unexpected non-continuation byte 0x7a, immediately after start byte 0xe3) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5948. Malformed UTF-8 character (unexpected continuation byte 0xba, with no preceding start byte) at /usr/lib/perl5/vendor_perl/5.8.0/Date/Manip.pm line 5948. Expected Results: Usually Date::Manip is quiet. It does appear to work correctly however. It's just really noisy. :-) Additional info:
A one-liner to reproduce this is: % perl -MDate::Manip -e 0 Expected result: no output. Actual result: all those malformed UTF-8 errors.
The same error messages are produced when building Date::Manip from CPAN, so it's not that the perl-DateManip package is bad in itself. Must be some interaction between Manip.pm and the way that your perl is built. Setting LANG=C stops the errors, see <http://www.ee.ethz.ch/~slist/mrtg/msg20204.html>. (It's possible that other modules than Date::Manip have the same problem, of course.)
It seems to be related to the choice of UTF-8 locales. So en_US.UTF8 causes the error messages, but the locale en_US doesn't. This must be because Manip.pm contains literal high-bit-set characters, and when perl is running in the UTF-8 locale it expects its source code to use UTF- 8 as well. Which may or may not be a good idea, but that's what it does. Suggested fix is to change Manip.pm to use chr(xxx) instead of literal characters. That would stop these warning messages, it wouldn't get Date::Manip to work correctly with UTF-8 input I think, but it's a start.
Thanks for the tip. It seems the non-UTF-8 string literals were inside comments, anyway. So as a fix, I just iconv'd the file to UTF-8 and no more errors/warnings. Blissfully quiet
A fixed source RPM for Date::Manip is at <http://atrpms.physik.fu-berlin.de/dist/rh9/perl-DateManip/>.
For cross-reference: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=105866#c2
*** Bug 88017 has been marked as a duplicate of this bug. ***
Created attachment 99440 [details] Script to find Perl Modules that iconv dislikes This script will give a listing of Perl Modules that iconv doesn't like. This might be useful for fixing other modules. I'll create another bug if people think this is useful.
Is "non-UTF-8 string literals were inside comments" still fatal in modern perl, and if so would something like the above script be useful in an automated test to avoid this problem in our future perl packages?
This is definitely no longer a bug with all current perl versions in all Red Hat linux releases, including the perl-5.8.0+ in RHEL-3 . o perl no longer complains about invalid UTF8 in comments o The Date::Manip module no longer contains invalid UTF-8 This bug can now be closed.