Bug 874633

Summary:	[abrt] openbox-3.5.0-3.fc16: openbox-xdg-menu:69:walk_menu:UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 26: ordinal not in range(128)
Product:	[Fedora] Fedora	Reporter:	Miroslav Lichvar <mlichvar>
Component:	openbox	Assignee:	Miroslav Lichvar <mlichvar>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	17	CC:	athmanem, christoph.wickert, claudiomar.costa, ejsheldrake, haliyo, ktim888, mlichvar, st.greb, tcallawa, vassari79, wili920503
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	799663	Environment:
Last Closed:	2012-12-07 03:31:00 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	799663
Bug Blocks:

Description Miroslav Lichvar 2012-11-08 14:59:07 UTC

+++ This bug was initially created as a clone of Bug #799663 +++
--- Additional comment from haliyo.fr on 2012-10-28 07:35:12 EDT ---

I'm still getting this on F17 KDE. LANG is fr_FR.UTF-8
I've already applied pyxdg fix from Bug 869877

[haliyo@pad ~]$  /usr/libexec/openbox-xdg-menu applications > app.menu
/usr/lib/python2.7/site-packages/xdg/Menu.py:1115: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if menuentry.DesktopFileID not in ids:
/usr/lib/python2.7/site-packages/xdg/Menu.py:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  """
Traceback (most recent call last):
  File "/usr/libexec/openbox-xdg-menu", line 101, in <module>
    map(walk_menu, menu.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 79, in walk_menu
    '<command>%s</command></action>' % command
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 173: ordinal not in range(128)
____________________________________________________________________________
[haliyo@pad ~]$ /usr/libexec/openbox-xdg-menu preferences > pref.menu
Traceback (most recent call last):
  File "/usr/libexec/openbox-xdg-menu", line 101, in <module>          
    map(walk_menu, menu.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 73, in walk_menu          
    command = re.sub(' -caption "%c"| -caption %c', ' -caption "%s"' % entry_name(entry.DesktopEntry), entry.DesktopEntry.getExec())
  File "/usr/lib64/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32: ordinal not in range(128)

--- Additional comment from haliyo.fr on 2012-10-28 07:35:53 EDT ---

Created attachment 634494 [details]
app.menu

--- Additional comment from haliyo.fr on 2012-10-28 07:36:17 EDT ---

Created attachment 634495 [details]
pref.menu

--- Additional comment from mlichvar on 2012-10-29 14:09:54 EDT ---

That's odd. Are you sure in both cases the locale is UTF-8?

Can you try to find the offending desktop file? Running it under "strace -eopen" might help.

--- Additional comment from haliyo.fr on 2012-10-29 17:14:21 EDT ---

(In reply to comment #16)
> That's odd. Are you sure in both cases the locale is UTF-8?

quite sure, yes:

[haliyo@pad ~]$ echo $LANG; /usr/libexec/openbox-xdg-menu applications > app.menu
fr_FR.UTF-8
/usr/lib/python2.7/site-packages/xdg/Menu.py:1115: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if menuentry.DesktopFileID not in ids:
/usr/lib/python2.7/site-packages/xdg/Menu.py:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  """
Traceback (most recent call last):
  File "/usr/libexec/openbox-xdg-menu", line 101, in <module>
    map(walk_menu, menu.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 79, in walk_menu
    '<command>%s</command></action>' % command
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 173: ordinal not in range(128)

> 
> Can you try to find the offending desktop file? Running it under "strace
> -eopen" might help.

will do that.

--- Additional comment from haliyo.fr on 2012-10-30 06:37:46 EDT ---

Hmm setting LANG=en_US.UTF-8 fixes for preferences but not for applications.

I couldn't tell the offending .desktop because in both cases (preferences en_US & fr_FR) there is no diff on any open.desktop.

I attach the logs hoping that you'll make make sense out of it than me.

--- Additional comment from haliyo.fr on 2012-10-30 06:39:06 EDT ---

Created attachment 635487 [details]
LANG=fr_Fr.UTF-8 LANGUAGE=fr_FR.UTF-8 strace -o strace_FR_prefs.txt -e open /usr/libexec/openbox-xdg-menu preferences > pref.menu

[haliyo@pad ~]$ LANG=fr_Fr.UTF-8 LANGUAGE=fr_FR.UTF-8 strace -o strace_FR_prefs.txt -e open /usr/libexec/openbox-xdg-menu preferences > pref.menu
Traceback (most recent call last):
  File "/usr/libexec/openbox-xdg-menu", line 101, in <module>
    map(walk_menu, menu.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 73, in walk_menu
    command = re.sub(' -caption "%c"| -caption %c', ' -caption "%s"' % entry_name(entry.DesktopEntry), entry.DesktopEntry.getExec())
  File "/usr/lib64/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32: ordinal not in range(128)

--- Additional comment from haliyo.fr on 2012-10-30 06:39:51 EDT ---

Created attachment 635488 [details]
LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 strace -o strace_EN_prefs.txt -e open /usr/libexec/openbox-xdg-menu preferences > pref.menu

[haliyo@pad ~]$ LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 strace -o strace_EN_prefs.txt -e open /usr/libexec/openbox-xdg-menu preferences > pref.menu
[haliyo@pad ~]$

--- Additional comment from haliyo.fr on 2012-10-30 06:41:31 EDT ---

Created attachment 635489 [details]
LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 strace -o strace_EN_apps.txt -e open /usr/libexec/openbox-xdg-menu applications > app.menu

Traceback (most recent call last):
  File "/usr/libexec/openbox-xdg-menu", line 101, in <module>
    map(walk_menu, menu.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 73, in walk_menu
    command = re.sub(' -caption "%c"| -caption %c', ' -caption "%s"' % entry_name(entry.DesktopEntry), entry.DesktopEntry.getExec())
  File "/usr/lib64/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32: ordinal not in range(128)
[haliyo@pad ~]$ LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 strace -o strace_EN_apps.txt -e open /usr/libexec/openbox-xdg-menu applications > app.menu
/usr/lib/python2.7/site-packages/xdg/Menu.py:1115: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if menuentry.DesktopFileID not in ids:
/usr/lib/python2.7/site-packages/xdg/Menu.py:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  """
Traceback (most recent call last):
  File "/usr/libexec/openbox-xdg-menu", line 101, in <module>
    map(walk_menu, menu.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 67, in walk_menu
    map(walk_menu, entry.getEntries())
  File "/usr/libexec/openbox-xdg-menu", line 79, in walk_menu
    '<command>%s</command></action>' % command
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 173: ordinal not in range(128)

--- Additional comment from mlichvar on 2012-10-30 12:52:11 EDT ---

Hm, I can't make any sense of the strace output either.

However, looking at your app.menu again, I see that the last (incomplete) entry is "Bibliothèque multimédia Microsoft", so I guess that's the offending entry for the application call.

The difference in behavior with different locale could be explained by different localized strings in the entries, like the name or icon.

--- Additional comment from ejsheldrake on 2012-10-30 15:14:09 EDT ---

I'm guessing the problem with the applications menu is due to the wine programs being installed into directories with non-ASCII characters in their names. This resulted in non-ASCII chars ending up in the value of the Exec key in the .desktop file. The desktop entry specification says that the value of the Exec key is a string (not a UTF-8 localestring) and so should be plain ASCII.

--- Additional comment from ejsheldrake on 2012-10-30 15:16:36 EDT ---

Created attachment 635724 [details]
example offending .desktop file

desktop-file-validate doesn't complain about this .desktop file.

--- Additional comment from ejsheldrake on 2012-10-30 15:19:28 EDT ---

Created attachment 635737 [details]
revised openbox-xdg-menu

It's openbox-xdg-menu with one more ".encode('utf-8')" thrown in.

--- Additional comment from mlichvar on 2012-10-31 05:28:47 EDT ---

Interesting, with your revised code it now fails here.

I see I'm still using the old pyxdg here. It seems the new pyxdg getExec() returns unicode instead of str. Isn't that a bug? I'm not sure it can assume in what encoding are the file names stored.

--- Additional comment from haliyo.fr on 2012-10-31 07:51:19 EDT ---

the revised version fixes it for me. Although it still gives a warning if run from console, the menu displays correctly on the desktop.
by the way, the sub-menu ids are still in english even if the lang is fr_Fr, but it's another bug.

[haliyo@pad ~]$ /usr/libexec/openbox-xdg-menu applications > app.menu
/usr/lib/python2.7/site-packages/xdg/Menu.py:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal                                            
  """
[haliyo@pad ~]$ 

thx!

Comment 1 Miroslav Lichvar 2012-11-08 16:03:03 UTC

I'm still not sure why getExec() sometimes returns unicode and sometimes str. I'd like to ask the pyxdg maintainer if it's not actually a bug in pyxdg before we implement a fix/workaround in xdg-menu.

Comment 2 Tom "spot" Callaway 2012-11-08 17:48:07 UTC

Go right ahead. :)

Comment 3 Fedora Update System 2012-11-13 16:43:37 UTC

openbox-3.5.0-9.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/openbox-3.5.0-9.fc17

Comment 4 Fedora Update System 2012-11-13 16:45:41 UTC

openbox-3.5.0-9.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/openbox-3.5.0-9.fc18

Comment 5 Fedora Update System 2012-11-15 02:38:13 UTC

Package openbox-3.5.0-9.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing openbox-3.5.0-9.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-18234/openbox-3.5.0-9.fc17
then log in and leave karma (feedback).

Comment 6 Miroslav Lichvar 2012-11-19 08:20:39 UTC

*** Bug 877798 has been marked as a duplicate of this bug. ***

Comment 7 Miroslav Lichvar 2012-11-26 14:27:11 UTC

*** Bug 879883 has been marked as a duplicate of this bug. ***

Comment 8 Miroslav Lichvar 2012-11-26 14:29:32 UTC

Please add karma to the update, so we can push it to stable. Thanks.

Comment 9 Miroslav Lichvar 2012-11-28 07:42:04 UTC

*** Bug 880808 has been marked as a duplicate of this bug. ***

Comment 10 Fedora Update System 2012-11-29 06:35:33 UTC

openbox-3.5.0-9.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 11 Fedora Update System 2012-12-07 03:31:03 UTC

openbox-3.5.0-9.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 12 Miroslav Lichvar 2012-12-07 07:37:06 UTC

*** Bug 884858 has been marked as a duplicate of this bug. ***

Comment 13 vassari 2012-12-07 12:32:02 UTC

After, and before, the update to openbox-3.5.0-9.fc17 I'm still getting this error with /usr/libexec/openbox-xdg-menu:

~ $ /usr/libexec/openbox-xdg-menu applications
Traceback (most recent call last):
  File "/usr/libexec/openbox-xdg-menu", line 100, in <module>
    menu = xdg.Menu.parse(menufile)
  File "/usr/lib/python2.7/site-packages/xdg/Menu.py", line 580, in parse
    __genmenuNotOnlyAllocated(tmp["Root"])
  File "/usr/lib/python2.7/site-packages/xdg/Menu.py", line 917, in __genmenuNotOnlyAllocated
    __genmenuNotOnlyAllocated(submenu)
  File "/usr/lib/python2.7/site-packages/xdg/Menu.py", line 920, in __genmenuNotOnlyAllocated
    tmp["cache"].addMenuEntries(menu.AppDirs)
  File "/usr/lib/python2.7/site-packages/xdg/Menu.py", line 1083, in addMenuEntries
    self.__addFiles(dir, "", prefix, legacy)
  File "/usr/lib/python2.7/site-packages/xdg/Menu.py", line 1089, in __addFiles
    menuentry = MenuEntry(os.path.join(subdir,item), dir, prefix)
  File "/usr/lib/python2.7/site-packages/xdg/Menu.py", line 425, in __init__
    self.DesktopEntry = DesktopEntry(os.path.join(dir,filename))
  File "/usr/lib/python2.7/site-packages/xdg/DesktopEntry.py", line 26, in __init__
    self.parse(filename)
  File "/usr/lib/python2.7/site-packages/xdg/DesktopEntry.py", line 34, in parse
    IniFile.parse(self, file, ["Desktop Entry", "KDE Desktop Entry"])
  File "/usr/lib/python2.7/site-packages/xdg/IniFile.py", line 54, in parse
    for line in fd:
  File "/usr/lib64/python2.7/codecs.py", line 296, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 178: invalid continuation byte


should I open a new bug?

Comment 14 Miroslav Lichvar 2012-12-07 13:08:29 UTC

Yes, please file a new bug, this seems to be something different.