Bug 1698514 - pungi-4.1.35-1.fc31 FTBFS with Python 3.8
Summary: pungi-4.1.35-1.fc31 FTBFS with Python 3.8
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: pungi
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Lubomír Sedlář
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: PYTHON38
TreeView+ depends on / blocked
 
Reported: 2019-04-10 14:07 UTC by Miro Hrončok
Modified: 2019-05-27 12:20 UTC (History)
7 users (show)

Fixed In Version: pungi-4.1.37-1.fc31
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-27 12:20:42 UTC
Type: Bug


Attachments (Terms of Use)
Full log from Copr (355.62 KB, text/plain)
2019-04-10 14:07 UTC, Miro Hrončok
no flags Details
Full log from Copr (Python 3.8.8a4) (417.24 KB, text/plain)
2019-05-15 11:57 UTC, Miro Hrončok
no flags Details

Description Miro Hrončok 2019-04-10 14:07:56 UTC
Created attachment 1554243 [details]
Full log from Copr

Python 3.8 changed how XML is sorted and pungi tests for exact string match in XML tests. This results in:

======================================================================
FAIL: test_filter_environments (tests.test_comps_wrapper.CompsWrapperTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/pungi-4.1.35/tests/test_comps_wrapper.py", line 85, in test_filter_environments
    self.assertFilesEqual(COMPS_ENVIRONMENT_FILE, self.file.name)
  File "/builddir/build/BUILD/pungi-4.1.35/tests/helpers.py", line 33, in assertFilesEqual
    self.assertEqual(diff, '', 'Files differ:\n' + diff)
AssertionError: '--- EXPECTED\n\n+++ ACTUAL\n\n@@ -10,7 +1[769 chars]mps>' != ''
Diff is 843 characters long. Set self.maxDiff to None to see it. : Files differ:
--- EXPECTED

+++ ACTUAL

@@ -10,7 +10,7 @@

     <default>true</default>
     <uservisible>true</uservisible>
     <packagelist>
-      <packagereq requires="dummy-imsettings" type="conditional">dummy-imsettings-gnome</packagereq>
+      <packagereq type="conditional" requires="dummy-imsettings">dummy-imsettings-gnome</packagereq>
     </packagelist>
   </group>
   <group>
@@ -99,8 +99,8 @@

     </grouplist>
   </environment>
   <langpacks>
-    <match install="aspell-%s" name="aspell"/>
-    <match install="firefox-langpack-%s" name="firefox"/>
-    <match install="kde-l10n-%s" name="kdelibs"/>
+    <match name="aspell" install="aspell-%s"/>
+    <match name="firefox" install="firefox-langpack-%s"/>
+    <match name="kdelibs" install="kde-l10n-%s"/>
   </langpacks>
 </comps>

======================================================================
FAIL: test_filter_groups (tests.test_comps_wrapper.CompsWrapperTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/pungi-4.1.35/tests/test_comps_wrapper.py", line 70, in test_filter_groups
    self.assertFilesEqual(COMPS_GROUP_FILE, self.file.name)
  File "/builddir/build/BUILD/pungi-4.1.35/tests/helpers.py", line 33, in assertFilesEqual
    self.assertEqual(diff, '', 'Files differ:\n' + diff)
AssertionError: '--- EXPECTED\n\n+++ ACTUAL\n\n@@ -10,7 +1[769 chars]mps>' != ''
Diff is 843 characters long. Set self.maxDiff to None to see it. : Files differ:
--- EXPECTED

+++ ACTUAL

@@ -10,7 +10,7 @@

     <default>true</default>
     <uservisible>true</uservisible>
     <packagelist>
-      <packagereq requires="dummy-imsettings" type="conditional">dummy-imsettings-gnome</packagereq>
+      <packagereq type="conditional" requires="dummy-imsettings">dummy-imsettings-gnome</packagereq>
     </packagelist>
   </group>
   <group>
@@ -88,8 +88,8 @@

     </grouplist>
   </environment>
   <langpacks>
-    <match install="aspell-%s" name="aspell"/>
-    <match install="firefox-langpack-%s" name="firefox"/>
-    <match install="kde-l10n-%s" name="kdelibs"/>
+    <match name="aspell" install="aspell-%s"/>
+    <match name="firefox" install="firefox-langpack-%s"/>
+    <match name="kdelibs" install="kde-l10n-%s"/>
   </langpacks>
 </comps>

======================================================================
FAIL: test_write_comps (tests.test_comps_wrapper.CompsWrapperTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/pungi-4.1.35/tests/test_comps_wrapper.py", line 60, in test_write_comps
    self.assertFilesEqual(COMPS_FORMATTED_FILE, self.file.name)
  File "/builddir/build/BUILD/pungi-4.1.35/tests/helpers.py", line 33, in assertFilesEqual
    self.assertEqual(diff, '', 'Files differ:\n' + diff)
AssertionError: '--- EXPECTED\n\n+++ ACTUAL\n\n@@ -10,7 +1[771 chars]mps>' != ''
Diff is 845 characters long. Set self.maxDiff to None to see it. : Files differ:
--- EXPECTED

+++ ACTUAL

@@ -10,7 +10,7 @@

     <default>true</default>
     <uservisible>true</uservisible>
     <packagelist>
-      <packagereq requires="dummy-imsettings" type="conditional">dummy-imsettings-gnome</packagereq>
+      <packagereq type="conditional" requires="dummy-imsettings">dummy-imsettings-gnome</packagereq>
     </packagelist>
   </group>
   <group>
@@ -122,8 +122,8 @@

     </grouplist>
   </environment>
   <langpacks>
-    <match install="aspell-%s" name="aspell"/>
-    <match install="firefox-langpack-%s" name="firefox"/>
-    <match install="kde-l10n-%s" name="kdelibs"/>
+    <match name="aspell" install="aspell-%s"/>
+    <match name="firefox" install="firefox-langpack-%s"/>
+    <match name="kdelibs" install="kde-l10n-%s"/>
   </langpacks>
 </comps>

----------------------------------------------------------------------
Ran 1025 tests in 7.574s

FAILED (SKIP=65, failures=3)

Full log attached.


See upstream (Python) issue:

https://bugs.python.org/issue34160
https://mail.python.org/pipermail/python-dev/2019-March/156709.html

Comment 1 Victor Stinner 2019-04-17 13:57:37 UTC
It's really hard to run pungi tests on Fedora Rawhide, since pungi setup.py and tox.ini omit many dependencies, and some dependencies are simply not available on Python 3.8... I used the following commands:
---
sudo dnf install -y python3-tox libxslt-devel libffi-devel krb5-devel python3-libcomps
git clone https://pagure.io/forks/vstinner/pungi.git
cd pungi
tox -e py38
# tox fails...
ln -s /usr/lib64/python3.7/site-packages/libcomps .tox/py38/lib/python3.8/site-packages/
.tox/py38/bin/python -m pip install nose koji PyYAML
LANG= .tox/py38/bin/nosetests tests/
---

I tried the following change but it doesn't fix the issue:

commit 14a4c33a4baadd77fad7d44fac34dda5c7aa5e40
Author: Victor Stinner <vstinner@redhat.com>
Date:   Wed Apr 17 14:26:29 2019 +0200

    Fix XML serialization on Python 3.8
    
    Python dict keeps insertion order since Python 3.6 and XML
    serialization no longer sort attributes by their name since Python
    3.8. So declare attributes in the order expected by tests (sort by
    name).
    
    Resolves: rhbz#1698514

diff --git a/pungi/dnf_wrapper.py b/pungi/dnf_wrapper.py
index af1d0cb..129b223 100644
--- a/pungi/dnf_wrapper.py
+++ b/pungi/dnf_wrapper.py
@@ -101,7 +101,7 @@ class CompsWrapper(object):
             packages.extend([i.name for i in group.optional_packages])
 
         for package in group.conditional_packages:
-            conditional.append({"name": package.requires, "install": package.name})
+            conditional.append({"install": package.name, "name": package.requires})
 
         return packages, conditional
 
@@ -129,7 +129,7 @@ class CompsWrapper(object):
     def get_langpacks(self):
         result = []
         for name, install in self.comps._i.langpacks.items():
-            result.append({"name": name, "install": install})
+            result.append({"install": install, "name": name})
         return result

Comment 2 Miro Hrončok 2019-04-17 14:10:33 UTC
Alternatively with the mock config from https://copr.fedorainfracloud.org/coprs/g/python/python3.8/

$ mock -r fedora-rawhide-x86_64-python38.cfg install 'python(abi) = 3.8' python3-devel python3-koji python3-libcomps python3-tox python3-lxml python3-pyyaml git-core
$ mock -r fedora-rawhide-x86_64-python38.cfg --enable-network shell
<mock-chroot> # git clone https://pagure.io/pungi.git
<mock-chroot> # cd pungi/
<mock-chroot> # python3 -m venv --system-site-packages __venv__
<mock-chroot> # . __venv__/bin/activate
(__venv__) <mock-chroot> # pip install nose mock
(__venv__) <mock-chroot> # pip install -e .
(__venv__) <mock-chroot> # nosetests tests/

That gives 258 errors (AttributeError: ... object has no attribute 'assertItemsEqual')

And 2 failures:

======================================================================
FAIL: test_filter_environments (tests.test_comps_wrapper.CompsWrapperTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/pungi/tests/test_comps_wrapper.py", line 85, in test_filter_environments
    self.assertFilesEqual(COMPS_ENVIRONMENT_FILE, self.file.name)
  File "/pungi/tests/helpers.py", line 33, in assertFilesEqual
    self.assertEqual(diff, '', 'Files differ:\n' + diff)
AssertionError: '--- EXPECTED\n\n+++ ACTUAL\n\n@@ -10,7 +1[769 chars]mps>' != ''
Diff is 843 characters long. Set self.maxDiff to None to see it. : Files differ:
--- EXPECTED

+++ ACTUAL

@@ -10,7 +10,7 @@

     <default>true</default>
     <uservisible>true</uservisible>
     <packagelist>
-      <packagereq requires="dummy-imsettings" type="conditional">dummy-imsettings-gnome</packagereq>
+      <packagereq type="conditional" requires="dummy-imsettings">dummy-imsettings-gnome</packagereq>
     </packagelist>
   </group>
   <group>
@@ -99,8 +99,8 @@

     </grouplist>
   </environment>
   <langpacks>
-    <match install="aspell-%s" name="aspell"/>
-    <match install="firefox-langpack-%s" name="firefox"/>
-    <match install="kde-l10n-%s" name="kdelibs"/>
+    <match name="aspell" install="aspell-%s"/>
+    <match name="firefox" install="firefox-langpack-%s"/>
+    <match name="kdelibs" install="kde-l10n-%s"/>
   </langpacks>
 </comps>

======================================================================
FAIL: test_filter_groups (tests.test_comps_wrapper.CompsWrapperTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/pungi/tests/test_comps_wrapper.py", line 70, in test_filter_groups
    self.assertFilesEqual(COMPS_GROUP_FILE, self.file.name)
  File "/pungi/tests/helpers.py", line 33, in assertFilesEqual
    self.assertEqual(diff, '', 'Files differ:\n' + diff)
AssertionError: '--- EXPECTED\n\n+++ ACTUAL\n\n@@ -10,7 +1[769 chars]mps>' != ''
Diff is 843 characters long. Set self.maxDiff to None to see it. : Files differ:
--- EXPECTED

+++ ACTUAL

@@ -10,7 +10,7 @@

     <default>true</default>
     <uservisible>true</uservisible>
     <packagelist>
-      <packagereq requires="dummy-imsettings" type="conditional">dummy-imsettings-gnome</packagereq>
+      <packagereq type="conditional" requires="dummy-imsettings">dummy-imsettings-gnome</packagereq>
     </packagelist>
   </group>
   <group>
@@ -88,8 +88,8 @@

     </grouplist>
   </environment>
   <langpacks>
-    <match install="aspell-%s" name="aspell"/>
-    <match install="firefox-langpack-%s" name="firefox"/>
-    <match install="kde-l10n-%s" name="kdelibs"/>
+    <match name="aspell" install="aspell-%s"/>
+    <match name="firefox" install="firefox-langpack-%s"/>
+    <match name="kdelibs" install="kde-l10n-%s"/>
   </langpacks>
 </comps>

======================================================================
FAIL: test_write_comps (tests.test_comps_wrapper.CompsWrapperTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/pungi/tests/test_comps_wrapper.py", line 60, in test_write_comps
    self.assertFilesEqual(COMPS_FORMATTED_FILE, self.file.name)
  File "/pungi/tests/helpers.py", line 33, in assertFilesEqual
    self.assertEqual(diff, '', 'Files differ:\n' + diff)
AssertionError: '--- EXPECTED\n\n+++ ACTUAL\n\n@@ -10,7 +1[771 chars]mps>' != ''
Diff is 845 characters long. Set self.maxDiff to None to see it. : Files differ:
--- EXPECTED

+++ ACTUAL

@@ -10,7 +10,7 @@

     <default>true</default>
     <uservisible>true</uservisible>
     <packagelist>
-      <packagereq requires="dummy-imsettings" type="conditional">dummy-imsettings-gnome</packagereq>
+      <packagereq type="conditional" requires="dummy-imsettings">dummy-imsettings-gnome</packagereq>
     </packagelist>
   </group>
   <group>
@@ -122,8 +122,8 @@

     </grouplist>
   </environment>
   <langpacks>
-    <match install="aspell-%s" name="aspell"/>
-    <match install="firefox-langpack-%s" name="firefox"/>
-    <match install="kde-l10n-%s" name="kdelibs"/>
+    <match name="aspell" install="aspell-%s"/>
+    <match name="firefox" install="firefox-langpack-%s"/>
+    <match name="kdelibs" install="kde-l10n-%s"/>
   </langpacks>
 </comps>

----------------------------------------------------------------------
Ran 924 tests in 4.146s

FAILED (SKIP=138, errors=258, failures=3)

Comment 3 Lubomír Sedlář 2019-04-18 08:27:05 UTC
Thank you for the testing guidance. With this patch [0] I can get all tests passing.

[0]: https://pagure.io/pungi/pull-request/1174

Comment 4 Miro Hrončok 2019-04-18 09:07:31 UTC
Thanks. Testing in copr build.

Comment 5 Miro Hrončok 2019-04-18 09:18:29 UTC
Works. Please close this bug once that code is part of the Fedora package. No need to patch the package immediately, we don't plan to move this forward until at least Python 3.8.0 beta 1 (2019-05-26).

Comment 6 Lubomír Sedlář 2019-04-18 11:15:14 UTC
The patch is merged and will go out in next release.

Comment 7 Miro Hrončok 2019-05-15 11:57:15 UTC
New failure with 3.8.0a4:

======================================================================
FAIL: test_validate_dummy_config (tests.test_config_validate_script.ConfigValidateScriptTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/pungi-4.1.36/tests/test_config_validate_script.py", line 30, in test_validate_dummy_config
    self.assertEqual(b'', stderr)
AssertionError: b'' != b'/usr/lib/python3.8/site-packages/requests[441 chars]e.\n'

----------------------------------------------------------------------
Ran 1033 tests in 10.572s

I believe this is some new DeprecationWarning on the stderr messing with the test.

Comment 8 Miro Hrončok 2019-05-15 11:57:44 UTC
Created attachment 1568974 [details]
Full log from Copr (Python 3.8.8a4)

Comment 9 Lubomír Sedlář 2019-05-15 13:54:31 UTC
You're right, the test trips on a warning. The warning is not coming from Pungi code, so I think the best short-term solution is to silence it. That being said, it's definitely a good idea to fix such errors where we can.

PR: https://pagure.io/pungi/pull-request/1192

Comment 10 Miro Hrončok 2019-05-17 14:47:09 UTC
I currently have this:

+Patch5:         https://pagure.io/pungi/pull-request/1174.patch
+Patch6:         https://pagure.io/pungi/pull-request/1192.patch

And the package builds.

Comment 11 Miro Hrončok 2019-05-24 06:23:17 UTC
You keep adding new patches to the spec, without applying those. Making me rebase again and again and again.

Could you please add the two patches as well?

Comment 12 Miro Hrončok 2019-05-27 12:20:42 UTC
Builds now. Thanks.


Note You need to log in before you can comment on or make changes to this bug.