RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1693712 - Switch to a string for RPM calls
Summary: Switch to a string for RPM calls
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: rpmlint
Version: 8.0
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: 8.0
Assignee: Thomas Woerner
QA Contact: Karel Srot
URL:
Whiteboard:
Depends On: 1631292 1715212
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-28 14:11 UTC by Panu Matilainen
Modified: 2020-11-14 12:23 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-05 21:13:21 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3437 0 None None None 2019-11-05 21:13:29 UTC

Description Panu Matilainen 2019-03-28 14:11:46 UTC
Description of problem:

Rpm's python3 API has been totally braindamaged all this time but people are only noticing now that it's starting to get used. 

We're changing rpm to return all string data as surrogate-escaped utf-8 python strings everywhere (instead of bytes with unknown encoding that the API doesn't otherwise even accept, see bug 1631292). This makes most rpm-scripts written for python2 just work with python3 too (from the rpm pov).

Most software that has kept python2 compatibility are automatically compatible with the fixed API, but unfortunately python3-only users like rpmlint need fixing for the new behavior.

There's at least one affected place in rpmlint, which will after the change start failing with the following traceback:

Traceback (most recent call last):
  File "/usr/bin/rpmlint", line 378, in <module>
    main()
  File "/usr/bin/rpmlint", line 166, in main
    runChecks(pkg)
  File "/usr/bin/rpmlint", line 223, in runChecks
    check.check(pkg)
  File "/usr/share/rpmlint/TagsCheck.py", line 695, in check
    self.check_summary(pkg, lang, ignored_words)
  File "/usr/share/rpmlint/TagsCheck.py", line 903, in check_summary
    if not Pkg.is_utf8_bytestr(summary):
  File "/usr/share/rpmlint/Pkg.py", line 168, in is_utf8_bytestr
    s.decode('UTF-8')
AttributeError: 'str' object has no attribute 'decode'

As the broken rpm versions are widely in use, it's best to keep compatibility with both initially. One possible way to fix this is simply:

--- Pkg.py.orig	2019-03-28 16:06:54.491218904 +0200
+++ Pkg.py	2019-03-28 16:07:13.412186582 +0200
@@ -168,6 +168,8 @@
         s.decode('UTF-8')
     except UnicodeError:
         return False
+    except AttributeError:
+        return True
     return True

Comment 1 Thomas Woerner 2019-06-13 12:54:06 UTC
The change in the 8.1 rpm requires to also change rpmlint. Without a fix the output of rpmlint will contain warning messages.

Here is a simple reproducer:

$ rpmlint systemd | grep UnicodeWarning
/var/str/source/Pkg.py:184: UnicodeWarning: decode() called on unicode string, see https://bugzilla.redhat.com/show_bug.cgi?id=1693751
  s.decode('UTF-8')
/var/str/source/Pkg.py:184: UnicodeWarning: decode() called on unicode string, see https://bugzilla.redhat.com/show_bug.cgi?id=1693751
  s.decode('UTF-8')

The proposed fix from #1693712#c0 is not working.

Here is a working proposal:

 def is_utf8_bytestr(s):
-    try:
-        s.decode('UTF-8')
-    except UnicodeError:
-        return False
-    return True
-
+    if isinstance(s, str):
+        try:
+            s.encode("utf8")
+        except UnicodeEncodeError:
+            return False
+        else:
+            return True
+    else:
+        try:
+            s.decode('UTF-8')
+        except UnicodeError:
+            return False
+        else:
+            return True

The encode in the first part is only there to make sure that we really have something that can be utf8. A str type in Python3 should always be utf8, but it is possible also to add bad escape sequences. This test is revealing this.

The output of the gating tests that will be added with #1682275 also contain lots of the warning lines.

Comment 2 Panu Matilainen 2019-06-13 13:01:51 UTC
Oh, sorry, the proposed code was prior to adding the .decode() compatibility kludge. Which once again is causing more problems than its solving. I think we need to reconsider the compat thing on rpm side...

Comment 11 errata-xmlrpc 2019-11-05 21:13:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3437


Note You need to log in before you can comment on or make changes to this bug.