If a line contains only diacritical characters (àéù for instance), function "length" returns the right number of caracters.

I have two RHEL servers version 7.3 (Maipo) with same version of awk (GNU Awk 4.0.2).  I have the problem on one but not the other!  Chances are that the problem lie with other part of the server or with its configuration.

Server with the problem:

oracle@slpgbd0221 :/oracle/tmp$ which awk
/bin/awk
PID:8203 OH:/oracle/product/11.2.0.4 -  SID:oemp(v.) - 18:15:20 - Nb err:3379
oracle@slpgbd0221 :/oracle/tmp$ ls -l /bin/awk
lrwxrwxrwx. 1 root root 4 Jun 15 12:02 /bin/awk -> gawk
PID:8203 OH:/oracle/product/11.2.0.4 -  SID:oemp(v.) - 18:15:27 - Nb err:3379
oracle@slpgbd0221 :/oracle/tmp$ sum /bin/awk
62099   419


Server without the problem:
$ which awk
/usr/bin/awk
$ ls -l  /usr/bin/awk
lrwxrwxrwx. 1 root root 4 Jun 15  2016 /usr/bin/awk -> gawk
$ sum /usr/bin/awk
62099   419

I can't reproduce the problem with vi anymore!  When I enter "unité" with vi, the octal dump is: "165 156 151 164 303 251 012".
When it comes from the file I was trying to process with awk and isolate "unité", octoal dump is "0000000 165 156 151 164 351 012".

I guess it has to do with UTF.  It is as if awk expects UTF and when it gets characters above 255, it skips them.

I realize, now, that both servers are plagged with the same bug.
When "unité" comes from vi, both servers can handle the "length" properly.  When "unité" comes from the file I was trying to process with awk, both servers return a length of 4 instead of 5.

I which I can upload the file.  I don't see where I can upload it to this bug.

Created attachment 1288481 [details]
Strip down version of the text file I am trying to process.  It contains only 5 characters (unité) and awk does not process them properly.

When I wrote "I guess it has to do with UTF.  It is as if awk expects UTF and when it gets characters above 255, it skips them" above, I meant "I guess it has to do with UTF.  It is as if awk expects UTF and when it gets characters above 127, it skips them."

I am trying to reproduce the file I am trying to process with awk and I could not.
awk 'BEGIN { X = sprintf("%c",233); print "unit" X }' | od -b
0000000 165 156 151 164 303 251 012

Character 233 (decimal) is converted in two bytes: 195 and 169.  In short, "awk" generates UTF8; I guess it expects UTF8 as input.  How can I make it process ISO8859-1 input streams?

I found it: one has to set environment variable LANG to C!

If the input data is not utf-8 encoded, you need to set locale accordingly.  You can try something like this:

$ export LANG=en_US.iso88591

It of course depends on which languages/encodings you are processing...

Hello Marc,

(In reply to Kamil Dudka from comment #11)
> If the input data is not utf-8 encoded, you need to set locale accordingly. 
> You can try something like this:
> 
> $ export LANG=en_US.iso88591
> 
> It of course depends on which languages/encodings you are processing...

Did Kamil's suggestion work for you?

Marc,

any update on your problem? :)

Hi David,

Unfortunately, I replied to your first EMail and, contrary to IBM support site, it did not get posted in this request.

Thanks for pointing out that "export LANG=en_US.io88591" solves the problem too.  I already knew that "export LANG=C" also solves the problem.

On AIX, my locale contains "en_US.ISO8859-1" and "en_US.8859-15", but not "en_US.iso88591".  On RHEL, I have "en_US.iso88591".

My goal is portability.  It seems that "C" locale exists on both platforms by default.  Maybe you have more portable solutions regarding this problem.

Otherwize, you can close that request.  However, I guess that I am not alone to get into that problem.  Maybe there is a way to keep track of this problem and its solution for future references.

Regards,
Marc.

The problem of plain text files is that they do not carry any (machine-readable) info about the encoding they use.  So it is nearly impossible to fix this problem in general.  Using LANG=C is the most portable setting but it hardly classifies as a (generic) solution to your problem.  While it may solve the problem with counting characters, as described in comment #0, it will not work well if you e.g. try to alphabetically sort the data.  The locale needs to be set properly.