Bug 1099096 - [abrt] UnicodeDecodeError from user input during ssh session
Summary: [abrt] UnicodeDecodeError from user input during ssh session
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 20
Hardware: i686
OS: Unspecified
low
unspecified
Target Milestone: ---
Assignee: Honza Silhan
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:f67a2b23220f2b00a350b2bbdc3...
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-05-19 13:34 UTC by Mikhail
Modified: 2015-06-30 01:01 UTC (History)
7 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-06-30 01:01:31 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: backtrace (1.38 KB, text/plain)
2014-05-19 13:35 UTC, Mikhail
no flags Details
File: environ (2.54 KB, text/plain)
2014-05-19 13:35 UTC, Mikhail
no flags Details
terminal encoding screenshot (98.00 KB, image/png)
2014-07-14 17:49 UTC, Mikhail
no flags Details

Description Mikhail 2014-05-19 13:34:54 UTC
Version-Release number of selected component:
dnf-0.5.1-1.fc20

Additional info:
reporter:       libreport-2.2.2
cmdline:        /usr/bin/python /usr/bin/dnf update
dso_list:       python-libs-2.7.5-11.fc20.i686
executable:     /usr/bin/dnf
kernel:         3.14.4-200.fc20.i686+PAE
runlevel:       N 5
type:           Python
uid:            0

Truncated backtrace:
utf_8.py:16:decode:UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

Traceback (most recent call last):
  File "/usr/bin/dnf", line 36, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 206, in user_main
    errcode = main(args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 82, in main
    return _main(base, args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 141, in _main
    ret = resolving(cli, base)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 169, in resolving
    return_code, resultmsgs = base.do_transaction()
  File "/usr/lib/python2.7/site-packages/dnf/cli/cli.py", line 211, in do_transaction
    if self.conf.assumeno or not self.output.userconfirm():
  File "/usr/lib/python2.7/site-packages/dnf/cli/output.py", line 943, in userconfirm
    choice = ucd(choice).lower()
  File "/usr/lib/python2.7/site-packages/dnf/i18n.py", line 129, in ucd
    return unicode(str(obj), _guess_encoding())
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 0: invalid continuation byte

Local variables in innermost frame:
input: <read-only buffer ptr 0x9a291cc, size 2 at 0x96816a0>
errors: 'strict'

Comment 1 Mikhail 2014-05-19 13:35:00 UTC
Created attachment 897148 [details]
File: backtrace

Comment 2 Mikhail 2014-05-19 13:35:03 UTC
Created attachment 897149 [details]
File: environ

Comment 3 Radek Holy 2014-05-19 15:04:34 UTC
Hello, thank you for the report. We'll take a look.

What was the answer to the transaction confirmation question that you typed?

Comment 4 Honza Silhan 2014-05-23 08:23:36 UTC
Hi, aside from your input, tell us what's your terminal encoding (for Gnome terminal it's located in terminal > Set character encoding), please. You have all enviromental LANG variables set to utf-8 but your string doesn't look like utf-8 encoded. Thanks.

Comment 5 Mikhail 2014-07-11 11:49:35 UTC
Another user experienced a similar problem:

Answer N in dnf

reporter:       libreport-2.2.2
cmdline:        /usr/bin/python /usr/bin/dnf update
dso_list:       python-libs-2.7.5-11.fc20.i686
executable:     /usr/bin/dnf
kernel:         3.14.4-200.fc20.i686+PAE
package:        dnf-0.5.1-1.fc20
reason:         utf_8.py:16:decode:UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 0: invalid continuation byte
runlevel:       N 5
type:           Python
uid:            0

Comment 6 Honza Silhan 2014-07-14 17:40:52 UTC
I tried to reproduce it with upstream and 0.5.1 version and I haven't seen any error.

(In reply to Jan Silhan from comment #4)
> tell us what's your terminal encoding (for Gnome
> terminal it's located in terminal > Set character encoding), please.

Does "y" answer works? Does Yum work with your mystical setup?

Comment 7 Mikhail 2014-07-14 17:49:57 UTC
Created attachment 917939 [details]
terminal encoding screenshot

Comment 8 Honza Silhan 2014-07-21 13:34:29 UTC
Thanks for the info, unfortunatelly it still doesn't help me to reproduce this bug. Atleast we know you have correct terminal encoding setup.

(In reply to Jan Silhan from comment #6)
> Does "y" answer works? Does Yum work with your mystical setup?
Can you also try that in dnf-0.5.4?

Comment 9 Honza Silhan 2014-08-27 16:21:07 UTC
Closing this because of insufficient information - can't reproduce.

Comment 10 Mikhail 2014-09-14 15:16:53 UTC
Again occurred today. When I answer 'Y' in remote ssh session.

# rpm -q dnf
dnf-0.5.4-2.fc20.noarch

Comment 11 Honza Silhan 2014-09-15 11:44:27 UTC
So it happened only in remote session? Tell us LANG and LC_ALL variables of the remote and local machine, please.

Comment 12 Mikhail 2014-09-15 13:44:06 UTC
[mikhail@localhost ~]$ echo $LC_ALL

[mikhail@localhost ~]$ echo $LANG
en_US.UTF-8
[mikhail@localhost ~]$ ssh root.1.113
root.1.113's password: 
Last failed login: Mon Sep 15 19:43:05 YEKT 2014 from 192.168.1.4 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Mon Sep 15 14:21:08 2014
[root@localhost ~]# echo $LC_ALL

[root@localhost ~]# echo $LANG
en_US.UTF-8

Comment 13 Mikhail 2014-09-15 17:23:35 UTC
And again I am catch this bug:


Upgraded:
  autocorr-en.noarch 1:4.2.6.3-3.fc20                             cpp.i686 4.8.3-7.fc20                                           gcc.i686 4.8.3-7.fc20                                                           
  gcc-c++.i686 4.8.3-7.fc20                                       libgomp.i686 4.8.3-7.fc20                                       libgcc.i686 4.8.3-7.fc20                                                        
  libstdc++-devel.i686 4.8.3-7.fc20                               libstdc++.i686 4.8.3-7.fc20                                     curl.i686 7.32.0-13.fc20                                                        
  libcurl.i686 7.32.0-13.fc20                                     gcc-base-debuginfo.i686 4.8.3-7.fc20                            gcc-debuginfo.i686 4.8.3-7.fc20                                                 
  iwl100-firmware.noarch 39.31.5.1-39.fc20                        iwl1000-firmware.noarch 1:39.31.5.1-39.fc20                     iwl105-firmware.noarch 18.168.6.1-39.fc20                                       
  iwl135-firmware.noarch 18.168.6.1-39.fc20                       iwl2000-firmware.noarch 18.168.6.1-39.fc20                      iwl2030-firmware.noarch 18.168.6.1-39.fc20                                      
  iwl3160-firmware.noarch 23.214.9.0-39.fc20                      iwl3945-firmware.noarch 15.32.2.9-39.fc20                       iwl4965-firmware.noarch 228.61.2.24-39.fc20                                     
  iwl5000-firmware.noarch 8.83.5.1_1-39.fc20                      iwl5150-firmware.noarch 8.24.2.2-39.fc20                        iwl6000-firmware.noarch 9.221.4.1-39.fc20                                       
  iwl6000g2a-firmware.noarch 17.168.5.3-39.fc20                   iwl6000g2b-firmware.noarch 17.168.5.2-39.fc20                   iwl6050-firmware.noarch 41.28.5.1-39.fc20                                       
  iwl7260-firmware.noarch 23.214.9.0-39.fc20                      java-1.7.0-openjdk.i686 1:1.7.0.65-2.5.2.5.fc20                 java-1.7.0-openjdk-headless.i686 1:1.7.0.65-2.5.2.5.fc20                        
  libasan.i686 4.8.3-7.fc20                                       libatomic.i686 4.8.3-7.fc20                                     libertas-usb8388-firmware.noarch 2:20140912-39.git365e80cce.fc20                
  libgfortran.i686 4.8.3-7.fc20                                   libquadmath.i686 4.8.3-7.fc20                                   libreoffice-calc.i686 1:4.2.6.3-3.fc20                                          
  libreoffice-core.i686 1:4.2.6.3-3.fc20                          libreoffice-ure.i686 1:4.2.6.3-3.fc20                           libreoffice-opensymbol-fonts.noarch 1:4.2.6.3-3.fc20                            
  libreoffice-writer.i686 1:4.2.6.3-3.fc20                        libreoffice-pdfimport.i686 1:4.2.6.3-3.fc20                     libreoffice-impress.i686 1:4.2.6.3-3.fc20                                       
  libreoffice-graphicfilter.i686 1:4.2.6.3-3.fc20                 libreoffice-draw.i686 1:4.2.6.3-3.fc20                          linux-firmware.noarch 20140912-39.git365e80cce.fc20                             
  pulseaudio.i686 5.0-7.fc20                                      pulseaudio-libs.i686 5.0-7.fc20                                 pulseaudio-module-x11.i686 5.0-7.fc20                                           
  pulseaudio-module-bluetooth.i686 5.0-7.fc20                     pulseaudio-utils.i686 5.0-7.fc20                                pulseaudio-libs-glib2.i686 5.0-7.fc20                                           
  pulseaudio-gdm-hooks.i686 5.0-7.fc20                            tzdata.noarch 2014g-1.fc20                                      tzdata-java.noarch 2014g-1.fc20                                                 
  xorg-x11-xinit.i686 1.3.4-1.fc20                               

Complete!
[root@localhost ~]# dnf downgrade pulseaudio pulseaudio-libs pulseaudio-module-x11 pulseaudio-module-bluetooth pulseaudio-utils pulseaudio-libs-glib2 pulseaudio-gdm-hooks
Dependencies resolved.

===================================================================================================================================================================================================================
 Package                                                         Arch                                     Version                                                 Repository                                  Size
===================================================================================================================================================================================================================
Downgrading:
 pulseaudio                                                      i686                                     4.0-8.gitf81e3.fc20                                     fedora                                     833 k
 pulseaudio-libs                                                 i686                                     4.0-8.gitf81e3.fc20                                     fedora                                     482 k
 pulseaudio-module-x11                                           i686                                     4.0-8.gitf81e3.fc20                                     fedora                                      26 k
 pulseaudio-module-bluetooth                                     i686                                     4.0-8.gitf81e3.fc20                                     fedora                                      48 k
 pulseaudio-utils                                                i686                                     4.0-8.gitf81e3.fc20                                     fedora                                      60 k
 pulseaudio-libs-glib2                                           i686                                     4.0-8.gitf81e3.fc20                                     fedora                                      15 k
 pulseaudio-gdm-hooks                                            i686                                     4.0-8.gitf81e3.fc20                                     fedora                                     7.3 k

Transaction Summary
===================================================================================================================================================================================================================
Downgrade  7 Packages

Total download size: 1.4 M
Is this ok [y/N]: y
Traceback (most recent call last):
  File "/usr/bin/dnf", line 36, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 200, in user_main
    errcode = main(args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 83, in main
    return _main(base, args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 140, in _main
    ret = resolving(cli, base)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 165, in resolving
    return_code, resultmsgs = base.do_transaction()
  File "/usr/lib/python2.7/site-packages/dnf/cli/cli.py", line 213, in do_transaction
    if self.conf.assumeno or not self.output.userconfirm():
  File "/usr/lib/python2.7/site-packages/dnf/cli/output.py", line 926, in userconfirm
    choice = ucd(choice).lower()
  File "/usr/lib/python2.7/site-packages/dnf/i18n.py", line 130, in ucd
    return unicode(str(obj), _guess_encoding())
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd0 in position 0: invalid continuation byte
[root@localhost ~]#

Comment 14 Mikhail 2014-09-15 17:26:45 UTC
Demonstration video: https://drive.google.com/file/d/0B0nwzlfiB4aQQUYzejRYUzF6ajg/edit?usp=sharing

I am do this in remote SSH session

Comment 15 Honza Silhan 2014-09-16 09:36:54 UTC
Nice content, subscribed! It's really mystery. Can you post output of:

$ python -c "import locale; print locale.getpreferredencoding()"
$ perl -Mcharnames=:full -CS -wle 'print "\N{EURO SIGN}"'

again locally and in remote session, please?

Comment 16 Mikhail 2014-09-16 09:41:35 UTC
[mikhail@localhost ~]$ python -c "import locale; print locale.getpreferredencoding()"
UTF-8
[mikhail@localhost ~]$ perl -Mcharnames=:full -CS -wle 'print "\N{EURO SIGN}"'
€
[mikhail@localhost ~]$ ssh root.1.113
root.1.113's password: 
Last login: Mon Sep 15 19:43:13 2014 from 192.168.1.4
[root@localhost ~]# python -c "import locale; print locale.getpreferredencoding()"
UTF-8
[root@localhost ~]# perl -Mcharnames=:full -CS -wle 'print "\N{EURO SIGN}"'
€
[root@localhost ~]#

Comment 17 Honza Silhan 2014-09-16 10:00:46 UTC
Thanks. I don't know what's going on. Everything seems to be set fine and you are the only one who experience this issue. Let's keep it open and wait for reports of other users to help us identify the cause.

Comment 18 Mikhail 2014-09-16 10:06:25 UTC
Demonstration video: https://drive.google.com/file/d/0B0nwzlfiB4aQYUxJcFhLMUh2Q3c/edit?usp=sharing

Could you try reproduce this?

Comment 19 Honza Silhan 2014-09-16 12:11:58 UTC
Summary from video:
==================
kyes        result
------------------
n<          x
nn<<        o
nn<         x
nnn         v
nnn<<<      x
nnnn<<<<    o
nnnn<<<     x
nnnn<<      o
n           o and v (wtf)
==================
o - empty str or unrecognized option
x - error
v - recognized option
n - 'n' char
< - del key

I think your terminal adds special chars when del key is pressed. I still cannot reproduce it.

Comment 20 Mikhail 2014-09-16 12:17:06 UTC
п is not  'n' char, this is not ASCII symbol. Cyrillic 'п'

Comment 21 Penelope Fudd 2015-05-12 07:35:31 UTC
Hello!

I just installed Fedora 22 KDE spin beta version, and updated all rpms.

When I type:
  dnf repoquery --qf '%{name} %{provides}' '*'
the program dies with the following error:

Traceback (most recent call last):
  File "/bin/dnf", line 36, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 185, in user_main
    errcode = main(args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 84, in main
    return _main(base, args)
  File "/usr/lib/python2.7/site-packages/dnf/cli/main.py", line 134, in _main
    cli.run()
  File "/usr/lib/python2.7/site-packages/dnf/cli/cli.py", line 1077, in run
    return self.command.run(self.base.extcmds)
  File "/usr/lib/python2.7/site-packages/dnf-plugins/repoquery.py", line 277, in run
    self.show_packages(q, fmt_fn)
  File "/usr/lib/python2.7/site-packages/dnf-plugins/repoquery.py", line 285, in show_packages
    print(fmt_fn(pkg))
  File "/usr/lib/python2.7/site-packages/dnf-plugins/repoquery.py", line 360, in provides
    return self._reldep_to_list(self._pkg.provides)
  File "/usr/lib/python2.7/site-packages/dnf-plugins/repoquery.py", line 332, in _reldep_to_list
    return '\n'.join([str(reldep) for reldep in obj])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 5: ordinal not in range(128)

I then tried
  dnf repoquery --qf '%{name} %{provides}' a*
and got the same error.

Then I started using the following to spell out the source of the problem:

  for n in a b c d e f g h i j k l m n o p q r s t u v w x y z; do dnf repoquery --qf '%{name} %{provides}' "a$n*" > /dev/null || echo $n failed ; done

  
I found that it was dying on 
  adobe-source-han-sans-twhk-fonts.noarch 
  adobe-source-han-sans-cn-fonts.noarch 
  baekmuk-ttf-hline-fonts.noarch
  baekmuk-ttf-gulim-fonts.noarch
  baekmuk-ttf-dotum-fonts.noarch
  baekmuk-ttf-batang-fonts.noarch
  
There's others, but I stopped here; you get the idea.

Comment 22 Penelope Fudd 2015-05-12 07:39:14 UTC
# python -c "import locale; print locale.getpreferredencoding()"
UTF-8

# perl -Mcharnames=:full -CS -wle 'print "\N{EURO SIGN}"'
€

# echo $LANG
en_US.UTF-8

# rpm -q dnf
dnf-1.0.0-1.fc22.noarch

# cat /etc/redhat-release
Fedora release 22 (Twenty Two)

Comment 23 Fedora End Of Life 2015-05-29 11:53:41 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 24 Fedora End Of Life 2015-06-30 01:01:31 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.