Bug 539904 - UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 9: ordinal not in range(128)
Summary: UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 9: ordin...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 13
Hardware: i386
OS: Linux
low
medium
Target Milestone: ---
Assignee: Ales Kozumplik
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: anaconda_trace_hash:42894e9d29388075f...
: 538328 (view as bug list)
Depends On:
Blocks: F13Beta, F13BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2009-11-21 14:46 UTC by dfinken
Modified: 2014-09-30 23:38 UTC (History)
25 users (show)

Fixed In Version: anaconda-13.36
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 575090 576237 (view as bug list)
Environment:
Last Closed: 2010-03-25 15:27:10 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Attached traceback automatically from anaconda. (41.47 KB, text/plain)
2009-11-21 14:46 UTC, dfinken
no flags Details
Attached traceback automatically from anaconda. (112.04 KB, text/plain)
2009-12-04 21:06 UTC, Joachim Schröder
no flags Details
Attached traceback automatically from anaconda. (41.14 KB, text/plain)
2010-01-03 21:15 UTC, jonathan
no flags Details
Evil hack to disable translation in system-config-date (848 bytes, patch)
2010-01-14 09:12 UTC, Thomas Jarosch
no flags Details | Diff
Attached traceback automatically from anaconda. (117.73 KB, text/plain)
2010-01-31 02:16 UTC, E Mair
no flags Details
Attached traceback automatically from anaconda. (71.40 KB, text/plain)
2010-02-21 13:55 UTC, Martin Naď
no flags Details
Attached traceback automatically from anaconda. (40.99 KB, text/plain)
2010-03-06 01:40 UTC, edgard
no flags Details
Attached traceback automatically from anaconda. (61.23 KB, text/plain)
2010-03-09 01:33 UTC, Didier G
no flags Details
Attached traceback automatically from anaconda. (110.73 KB, text/plain)
2010-03-09 19:34 UTC, Thomas Canniot
no flags Details
Attached traceback automatically from anaconda. (109.83 KB, text/plain)
2010-03-11 07:49 UTC, Milan Kerslager
no flags Details
Attached traceback automatically from anaconda. (62.87 KB, text/plain)
2010-03-17 20:29 UTC, Milan Kerslager
no flags Details
Attached traceback automatically from anaconda. (168.96 KB, text/plain)
2010-03-19 12:29 UTC, Ales Kozumplik
no flags Details
Attached traceback automatically from anaconda. (59.15 KB, text/plain)
2010-03-22 10:44 UTC, stellarboy653
no flags Details
Attached traceback automatically from anaconda. (150.12 KB, text/plain)
2010-03-30 08:59 UTC, Răzvan Sandu
no flags Details
Attached traceback automatically from anaconda. (142.53 KB, text/plain)
2010-04-07 13:25 UTC, Terje Røsten
no flags Details
Attached traceback automatically from anaconda. (64.57 KB, text/plain)
2010-04-11 12:59 UTC, ultima.ratio.regum69
no flags Details
Attached traceback automatically from anaconda. (88.09 KB, text/plain)
2010-04-26 09:11 UTC, Terje Røsten
no flags Details
Attached traceback automatically from anaconda. (42.58 KB, text/plain)
2010-05-09 10:40 UTC, jambus85
no flags Details

Description dfinken 2009-11-21 14:46:38 UTC
The following was filed automatically by anaconda:
anaconda 12.46 exception report
Traceback (most recent call first):
  File "/usr/lib/python2.6/site-packages/scdate/core/zonetab.py", line 81, in <lambda>
    translated = reduce(lambda x, y: x.replace(y, "/"),
  File "/usr/lib/python2.6/site-packages/scdate/core/zonetab.py", line 83, in __translate_tz
    translated)
  File "/usr/lib/python2.6/site-packages/scdate/core/zonetab.py", line 105, in _set_tz
    self.__translate_tz()
  File "/usr/lib/python2.6/site-packages/scdate/core/zonetab.py", line 51, in __init__
    self.tz = tz.replace ('_', ' ')
  File "/usr/lib/python2.6/site-packages/scdate/core/zonetab.py", line 196, in readZoneTab
    entry = ZoneTabEntry (code, lat, long, tz, comments)
  File "/usr/lib/python2.6/site-packages/scdate/core/zonetab.py", line 131, in __init__
    self.readZoneTab (fn)
  File "/usr/lib/anaconda/textw/timezone_text.py", line 39, in getTimezoneList
    zt = zonetab.ZoneTab()
  File "/usr/lib/anaconda/textw/timezone_text.py", line 67, in __call__
    timezones = self.getTimezoneList()
  File "/usr/lib/anaconda/text.py", line 480, in run
    rc = win(self.screen, instance)
  File "/usr/bin/anaconda", line 968, in <module>
    anaconda.intf.run(anaconda)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 9: ordinal not in range(128)

Comment 1 dfinken 2009-11-21 14:46:44 UTC
Created attachment 372746 [details]
Attached traceback automatically from anaconda.

Comment 2 Felix Kaechele 2009-11-23 11:54:23 UTC
This happened to me when I tried to kickstart a VM using cobbler.
It only happens to me when the system locale is set to de_DE.

Comment 3 Neustradamus 2009-11-24 04:27:13 UTC
same problem with french language (Fedora 12 i386 DVD .iso)

Comment 4 Chris 2009-11-24 13:59:19 UTC
same here with: lang de_DE.UTF-8

Comment 5 Joachim Schröder 2009-12-04 21:06:17 UTC
Created attachment 376204 [details]
Attached traceback automatically from anaconda.

Comment 6 Joachim Schröder 2009-12-04 21:28:27 UTC
this bug depends on the text mode for installation; using the graphical installer shows no problem with german or other locale, only using text mode (simply add "text" to the boot params) will throw this traceback when selecting german locale, e.g.
This was tested on two different machines, with the same result when using text mode.

Comment 7 Edgar Hoch 2009-12-07 21:04:53 UTC
I can confirm that anaconda crashes with kickstart options "text" and "lang de_DE".
It works proberly without kickstart options "text".

The reason for using text mode in previous fedora versions were problems with graphical mode with some graphic cards. By using text mode we always see an output on the display during installation, in contrast to a black or unreadable output with some graphic cards.

Is there a chance to create an update script for kickstart installation (to put in an update directory on the install image)?

Comment 8 jonathan 2010-01-03 21:15:22 UTC
Created attachment 381427 [details]
Attached traceback automatically from anaconda.

Comment 9 Thomas Jarosch 2010-01-14 09:12:54 UTC
Created attachment 383638 [details]
Evil hack to disable translation in system-config-date

I've created an evil hack patch that disables the translation stuff in the system-config-date package. 

Using this patch I'm able to spin a german install CD in text mode.
Too bad I'm not a python developer to fix this properly :(

Comment 10 Neustradamus 2010-01-17 20:18:57 UTC
you can see this: https://bugzilla.redhat.com/show_bug.cgi?id=538328

Comment 11 E Mair 2010-01-31 02:16:57 UTC
Created attachment 387800 [details]
Attached traceback automatically from anaconda.

Comment 12 Martin Naď 2010-02-21 13:55:13 UTC
Created attachment 395342 [details]
Attached traceback automatically from anaconda.

Comment 13 edgard 2010-03-06 01:40:36 UTC
Created attachment 398169 [details]
Attached traceback automatically from anaconda.

Comment 14 Didier G 2010-03-09 01:33:39 UTC
Created attachment 398660 [details]
Attached traceback automatically from anaconda.

Comment 15 Thomas Canniot 2010-03-09 19:34:14 UTC
Created attachment 398904 [details]
Attached traceback automatically from anaconda.

Comment 16 Milan Kerslager 2010-03-11 07:49:55 UTC
Created attachment 399273 [details]
Attached traceback automatically from anaconda.

Comment 17 Milan Kerslager 2010-03-11 07:51:40 UTC
Same here with Czech language (cs_CZ).

Comment 18 Milan Kerslager 2010-03-17 20:29:25 UTC
Created attachment 400878 [details]
Attached traceback automatically from anaconda.

Comment 19 Milan Kerslager 2010-03-17 20:34:21 UTC
13-Beta.TC0 still has the problem. Installation in text interface is impossible in other languages. Only English language works.

It seems like candidate for blocking bug for Beta.

Comment 20 Felix Kaechele 2010-03-17 21:13:25 UTC
Added as F13Beta blocker bug and changed Version to 13.
This bug essentially prevents any non-English Textmode Installs

Comment 21 Ales Kozumplik 2010-03-18 14:21:29 UTC
Here's what's *probably* going on:
in s-c-date, in zonetab.py, this code is called:
  translated = reduce(lambda x, y: x.replace(y, "/"), self._slash_lookalikes,   translated)
where translated is of type string and _slash_lookalikes is a list of unicodes. replace needs to decode translated into a unicode so the replacing can take place. it has no clue what translated's encoding is but it assumes it's the one from sys.getdefaultencoding() and that makes perfect sense. Unfortunately in anaconda we never set this to antyhing, so it's ascii, thus the traceback (i.e. python knows it's looking at an encoded unicode but at the same time it is not ascii so it bombs).

Why it does not happen during graphical install:
because we do 'import gtk' which has a nasty side-effect of setting the default encoding to 'utf8', see: https://bugzilla.gnome.org/show_bug.cgi?id=132040
BTW, the exact same thing happens when s-c-date is called standalone on an installed system.

Cut the story short and get to the root cause:
The default encoding in python is ascii during text install and not utf8.

What the fix is going to be like:
1) if we can figure out a way to have sys.setdefaultencoding('utf8') called before anaconda starts we're golden.
2) otherwise we would need to import gtk, even in text install, which would be strange.

Comment 22 Ales Kozumplik 2010-03-19 12:29:27 UTC
Created attachment 401231 [details]
Attached traceback automatically from anaconda.

Comment 23 Chris Lumens 2010-03-19 15:22:40 UTC
Ales, you've got a patch that looks like it's going to fix this problem so I am going to assign the bug to you.

Comment 24 Nils Philippsen 2010-03-22 10:04:17 UTC
(In reply to comment #21)
> 1) if we can figure out a way to have sys.setdefaultencoding('utf8') called
> before anaconda starts we're golden.

Hm, shouldn't it be sufficient to do "import sys; sys.setdefaultencoding('utf8')" right at the top of anaconda?

Comment 25 Ales Kozumplik 2010-03-22 10:26:43 UTC
Nils,

It's not that simple, unfortunately, from the setdefaultencoding documentation:
This function is only intended to be used by the site module implementation and, where needed, by sitecustomize. Once used by the site module, it is removed from the sys module’s namespace.

I have a patch that should achieve what we want, it's waiting here:
https://www.redhat.com/archives/anaconda-devel-list/2010-March/msg00284.html

Since it could have side effects we are talking to the python maintainers to give us a blessing.

Ales

Comment 26 stellarboy653 2010-03-22 10:44:53 UTC
Created attachment 401697 [details]
Attached traceback automatically from anaconda.

Comment 27 Nils Philippsen 2010-03-22 11:44:46 UTC
(In reply to comment #25)
> It's not that simple, unfortunately, from the setdefaultencoding documentation:
> This function is only intended to be used by the site module implementation
> and, where needed, by sitecustomize. Once used by the site module, it is
> removed from the sys module’s namespace.

I'm puzzled about how it works with gtk then...

Comment 28 Ales Kozumplik 2010-03-22 14:50:47 UTC
For F13, Fixed by e5b1f8c2eda67299a6abe1fc39a9756d3284b987.
For rhel6 please see Bug 575090.

Comment 29 Ales Kozumplik 2010-03-22 14:54:24 UTC
Nils,

they have a hack in place that overrides the inmutable python variable: https://bugzilla.gnome.org/show_bug.cgi?id=132040.

We just did something superficially cleaner and not as sneaky.

Ales

Comment 30 Nils Philippsen 2010-03-22 15:34:44 UTC
(In reply to comment #29)
> Nils,
> 
> they have a hack in place that overrides the inmutable python variable:
> https://bugzilla.gnome.org/show_bug.cgi?id=132040.

Yuck.
 
> We just did something superficially cleaner and not as sneaky.

I don't get it: There must be something cleaner to fix this, at least from the scdate angle. Would it using unicode vars instead of strs help here?

Comment 31 Nils Philippsen 2010-03-22 16:09:21 UTC
I found that creating ZoneTab objects without importing gtk gives similar tracebacks. Since I only ever used this in a gtk context, I never saw these problems by myself since it made unicode conversions from utf-8 magically work. If I change the following in zonetab.py, it works without using sys.setdefaultencoding() -- I tried it with de_DE.UTF-8 and fr_FR.UTF-8:

diff --git a/src/scdate/core/zonetab.py b/src/scdate/core/zonetab.py
index 1128bd7..4d7d994 100644
--- a/src/scdate/core/zonetab.py
+++ b/src/scdate/core/zonetab.py
@@ -29,7 +29,7 @@ import locale
 import warnings
 
 from gettext import ldgettext
-gettext = lambda x: ldgettext("system-config-date", x)
+ugettext = lambda x: unicode(ldgettext("system-config-date", x), "utf-8")
 
 __all__ = ("ZoneTabEntry", "ZoneTab")
 
@@ -56,7 +56,7 @@ class ZoneTabEntry(object):
             return
 
         # time zone msgid and msgstr have underscores instead of spaces
-        translated = gettext(self._tz.replace(" ", "_")).replace("_", " ")
+        translated = ugettext(self._tz.replace(" ", "_")).replace("_", " ")
 
         (lang, coding) = locale.getlocale()
         if lang is None:

Do you think I should rather apply this to s-c-date rather than you doing nasty things in anaconda?

Comment 32 Ales Kozumplik 2010-03-23 07:18:18 UTC
Yes,

having the strings in unicode from the start is a fix too and you should commit this patch to s-c-date.

To me it made sense to have anaconda start with the same default encoding every time, no matter whether it's running in text or gui, to prevent this kind of bugs. Chris, what is your opinion, should we keep the sitecustomize.py patch?

Ales

Comment 33 Nils Philippsen 2010-03-23 10:27:43 UTC
(In reply to comment #32)
> having the strings in unicode from the start is a fix too and you should commit
> this patch to s-c-date.

already committed to git, will be in the next version

> To me it made sense to have anaconda start with the same default encoding every
> time, no matter whether it's running in text or gui, to prevent this kind of
> bugs. Chris, what is your opinion, should we keep the sitecustomize.py patch?

Chris mentioned to me on IRC yesterday that he would like to keep it since you had more problems similar to this with other "3rd party" software. Perhaps you could compile a list of affected components and ping their respective maintainers? They should fix their code as well to work without importing gtk and its side-effects.

Thanks for your efforts in this.

Comment 34 Chris Lumens 2010-03-23 14:18:14 UTC
We've hit this problem in many places throughout anaconda, most notably dealing with yum and rpm, so I am fine with us keeping the patch in anaconda to prevent other problems from showing up in anaconda too.

Comment 35 Fedora Update System 2010-03-23 16:25:28 UTC
system-config-date-1.9.55-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/system-config-date-1.9.55-1.fc13

Comment 36 Fedora Update System 2010-03-23 21:29:44 UTC
anaconda-13.36-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/anaconda-13.36-1.fc13

Comment 37 Fedora Update System 2010-03-24 00:48:01 UTC
anaconda-13.36-1.fc13 has been pushed to the Fedora 13 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update anaconda'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/anaconda-13.36-1.fc13

Comment 38 Adam Williamson 2010-03-24 18:31:49 UTC
Hey, folks - can people who ran into this bug please re-test with F13 Beta TC1 - http://serverbeach1.fedoraproject.org/pub/alt/stage/13-Beta.TC1/ ? It should be fixed there, as that includes anaconda 13.36. Thanks!

Comment 39 Adam Williamson 2010-03-25 01:19:46 UTC
I tested TC1 here. With the DVD, doing a text mode install, I was able to successfully complete an installation selecting German as the language.

I'm not going to close this as I didn't see the initial issue so I can't be 100% sure I'm following the same method as the reporters, but it seems like a good result.

Comment 40 Milan Kerslager 2010-03-25 12:45:48 UTC
Seems to me like it has been solved with TC1.

Even there is another problem with font. Unicode (national) characters are brighter than normal characters (filled as bug #576845).

Comment 41 Adam Williamson 2010-03-25 15:27:10 UTC
Thanks, Milan - closing. If anyone still sees the bug in TC1, reopen (if you can) or post a comment and I'll reopen.

Comment 42 Răzvan Sandu 2010-03-30 08:59:06 UTC
Created attachment 403441 [details]
Attached traceback automatically from anaconda.

Comment 43 Terje Røsten 2010-04-07 13:25:01 UTC
Created attachment 404951 [details]
Attached traceback automatically from anaconda.

Comment 44 ultima.ratio.regum69 2010-04-11 12:59:04 UTC
Created attachment 405812 [details]
Attached traceback automatically from anaconda.

Comment 45 Adam Williamson 2010-04-12 17:52:58 UTC
for those who've hit this bug (Razvan, Terje, ultima.ratio.regum) - as you can see, it's fixed in later versions. If you pick up Fedora 13 Beta (which will be out shortly), it'll work fine there. Sorry for the inconvenience.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 46 Terje Røsten 2010-04-13 07:28:38 UTC
I tried a RC Beta release and that worked fine, thanks.

Comment 47 Fedora Update System 2010-04-16 23:42:21 UTC
system-config-date-1.9.56-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 48 Terje Røsten 2010-04-26 09:11:38 UTC
Created attachment 409118 [details]
Attached traceback automatically from anaconda.

Comment 49 Terje Røsten 2010-04-26 09:44:49 UTC
Comment #48 was from the boot.iso.

Comment 50 jambus85 2010-05-09 10:40:07 UTC
Created attachment 412625 [details]
Attached traceback automatically from anaconda.

Comment 51 Nils Philippsen 2010-08-24 15:32:26 UTC
*** Bug 538328 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.