Bug 1693759

Summary: mock: Switch to a string for RPM calls (oncoming rpm API fix change)
Product: [Fedora] Fedora Reporter: Panu Matilainen <pmatilai>
Component: mockAssignee: Miroslav Suchý <msuchy>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: jdisnard, jkeating, mebrown, msuchy, praiskup, williams
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mock-1.4.15-1.fc30 mock-1.4.15-1.fc28 mock-1.4.15-1.fc29 mock-1.4.15-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-27 21:27:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1693751, 1779194    

Description Panu Matilainen 2019-03-28 15:06:10 UTC
Description of problem:

Rpm's python3 API has been totally braindamaged all this time but people are only noticing now that it's starting to get used. 

We're changing rpm to return all string data as surrogate-escaped utf-8 python strings everywhere (instead of bytes with unknown encoding that the API doesn't otherwise even accept, see bug 1631292). This makes most rpm-scripts written for python2 just work with python3 too (from the rpm pov).

Most software that has kept python2 compatibility are automatically compatible with the fixed API, but unfortunately python3-only users like mock need fixing for the new behavior.

There's at least one affected place in mock, resulting in a traceback similar to this with the new behavior, unless fixed:

Traceback (most recent call last):
  File "/usr/libexec/mock/mock", line 977, in <module>
    main()
  File "/usr/lib/python3.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 766, in main
    run_command(options, args, config_opts, commands, buildroot, state)
  File "/usr/lib/python3.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 861, in run_command
    do_rebuild(config_opts, commands, buildroot, options, args)
  File "/usr/lib/python3.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 541, in do_rebuild
    post=post_build, clean=clean)
  File "/usr/lib/python3.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/libexec/mock/mock", line 482, in rebuild_generic
    ret = cmd(item)
  File "/usr/libexec/mock/mock", line 522, in build
    check=config_opts['check'], spec=options.spec)
  File "/usr/lib/python3.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.7/site-packages/mockbuild/backend.py", line 275, in build
    self.installSrpmDeps(rebuilt_srpm)
  File "/usr/lib/python3.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.7/site-packages/mockbuild/backend.py", line 198, in installSrpmDeps
    deps.extend(util.getAddtlReqs(hdr, self.more_buildreqs))
  File "/usr/lib/python3.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.7/site-packages/mockbuild/util.py", line 400, in getAddtlReqs
    (name, epoch, ver, rel, arch) = getNEVRA(hdr)
  File "/usr/lib/python3.7/site-packages/mockbuild/trace_decorator.py", line 96, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.7/site-packages/mockbuild/util.py", line 386, in getNEVRA
    return tuple(x.decode() if i != 1 else x for i, x in enumerate(ret))
  File "/usr/lib/python3.7/site-packages/mockbuild/util.py", line 386, in <genexpr>
    return tuple(x.decode() if i != 1 else x for i, x in enumerate(ret))
AttributeError: 'str' object has no attribute 'decode'

As broken versions of the rpm bindings are widely in use, it's recommended that compatibility with both versions is kept for the time being. A possible simple fix is just:
--- util.py.orig	2019-03-28 16:32:22.378591934 +0200
+++ util.py	2019-03-28 16:32:58.009530429 +0200
@@ -342,6 +342,10 @@
     for dummy in yieldSrpmHeaders(srpms, plainRpmOk):
         pass
 
+def b2s(data):
+    if isinstance(data, bytes):
+        return data.decode('utf-8')
+    return data
 
 @traceLog()
 def getNEVRA(hdr):
@@ -354,7 +358,7 @@
     if epoch is None:
         epoch = 0
     ret = (name, epoch, ver, rel, arch)
-    return tuple(x.decode() if i != 1 else x for i, x in enumerate(ret))
+    return tuple(b2s(x) if i != 1 else x for i, x in enumerate(ret))
 

Alternatively, upstream mock version already has a suitable helper added in commit 08b6e4a0eaf7e8317bbef4877302528cb60e6ea5, so the upstream fix would be simply:

--- a/mock/py/mockbuild/util.py
+++ b/mock/py/mockbuild/util.py
@@ -383,7 +383,7 @@ def getNEVRA(hdr):
     if epoch is None:
         epoch = 0
     ret = (name, epoch, ver, rel, arch)
-    return tuple(x.decode() if i != 1 else x for i, x in enumerate(ret))
+    return tuple(_to_text(x) if i != 1 else x for i, x in enumerate(ret))

Comment 1 Panu Matilainen 2019-03-29 07:10:04 UTC
I was pointed out that the bug referred to in the message is a private RHEL bug, sorry about that. 
The public, Fedora side counterpart with the background story is the one blocked by this, ie 
https://bugzilla.redhat.com/show_bug.cgi?id=1693751

Comment 2 Miroslav Suchý 2019-04-19 18:12:17 UTC
Committed as:
  * c7ce1fc (HEAD -> devel) switch to string rpm's API [RHBZ#1693759]

Comment 3 Fedora Update System 2019-04-22 19:13:28 UTC
mock-1.4.15-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-7a7a12aff5

Comment 4 Fedora Update System 2019-04-22 19:13:34 UTC
mock-1.4.15-1.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-98a73907d3

Comment 5 Fedora Update System 2019-04-22 19:13:40 UTC
mock-1.4.15-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-9f8e99528d

Comment 6 Fedora Update System 2019-04-22 19:13:46 UTC
mock-1.4.15-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0649a04cab

Comment 7 Fedora Update System 2019-04-23 14:55:50 UTC
mock-1.4.15-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-98a73907d3

Comment 8 Fedora Update System 2019-04-23 19:33:43 UTC
mock-1.4.15-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0649a04cab

Comment 9 Fedora Update System 2019-04-23 21:15:57 UTC
mock-1.4.15-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-7a7a12aff5

Comment 10 Fedora Update System 2019-04-23 23:40:47 UTC
mock-1.4.15-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-9f8e99528d

Comment 11 Fedora Update System 2019-04-27 21:27:41 UTC
mock-1.4.15-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 12 Fedora Update System 2019-04-30 01:40:32 UTC
mock-1.4.15-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Comment 13 Fedora Update System 2019-04-30 02:27:40 UTC
mock-1.4.15-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 14 Fedora Update System 2019-05-03 03:02:48 UTC
mock-1.4.15-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.