Bug 2079622

FEDORA-2022-2da3a47337 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-d6e6c7c37c has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2022-671a5d63e6 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

rhel-system-roles-1.18.0-1.el9.noarch was used.

There were some hiccups in testing and failed downloads, All the VMs seem to have worked now though in the Upstream-testsuite except RHEL 6. The results follow:

[root@intel-wildcatpass-02 Upstream-testsuite]# uname -r
5.14.0-96.el9.x86_64
[root@intel-wildcatpass-02 Upstream-testsuite]# rpm -q rhel-system-roles
rhel-system-roles-1.18.0-1.el9.noarch

:: [ 17:08:11 ] :: [  BEGIN   ] :: Test network/tests_routing_rules_nm.yml (roles/rhel-system-roles.network/te\
sts/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_6_10_GA :: actually running 'ansible-playbook -v\
v -i /usr/share/ansible/inventory/standard-inventory-qcow2 --skip-tags tests::nvme  _setup.yml roles/rhel-syst\
em-roles.network/tests/tests_routing_rules_nm.yml &> SYSTEM-ROLE-network_tests_routing_rules_nm.yml-legacy-ANS\
IBLE-2.9-RHEL_6_10_GA.log'
xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : Local character set is now assumed as: 'utf-8'
[ERROR] standard-inventory-qcow2: Fatal error in provision script.
Traceback (most recent call last):
  File "/usr/share/ansible/inventory/standard-inventory-qcow2", line 786, in <module>
    main()
  File "/usr/share/ansible/inventory/standard-inventory-qcow2", line 778, in main
    data = inv_list(opts)
  File "/usr/share/ansible/inventory/standard-inventory-qcow2", line 241, in inv_list
    host_vars = inv_host(opts, subject, hostalias)
  File "/usr/share/ansible/inventory/standard-inventory-qcow2", line 590, in inv_host
    data = f.readlines()
  File "/usr/lib64/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
:: [ 17:43:41 ] :: [   FAIL   ] :: Test network/tests_routing_rules_nm.yml (roles/rhel-system-roles.network/te\
sts/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_6_10_GA (Expected 0, got 2)


:: [ 17:57:07 ] :: [   PASS   ] :: Test network/tests_routing_rules_nm.yml (roles/rhel-system-roles.network/te\
sts/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_7_9_GA (Expected 0, got 0)
:: [ 18:04:06 ] :: [   PASS   ] :: Test network/tests_routing_rules_nm.yml (roles/rhel-system-roles.network/te\
sts/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_5_0_GA (Expected 0, got 0)
:: [ 18:07:39 ] :: [   PASS   ] :: Test network/tests_routing_rules_nm.yml (roles/rhel-system-roles.network/te\
sts/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_6_TESTING (Expected 0, got 0)
:: [ 18:37:58 ] :: [   PASS   ] :: Test network/tests_routing_rules_nm.yml (roles/rhel-system-roles.network/te\
sts/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_7_TESTING (Expected 0, got 0)
:: [ 18:14:18 ] :: [   PASS   ] :: Test network/tests_routing_rules_nm.yml (roles/rhel-system-roles.network/te\
sts/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_9_0_TESTING (Expected 0, got 0)
:: [ 18:18:02 ] :: [   PASS   ] :: Test network/tests_routing_rules_nm.yml (roles/rhel-system-roles.network/te\
sts/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_9_1_TESTING (Expected 0, got 0)

There seems to be a crypto problem:

tail /var/tmp/SYSTEM-ROLE-network_tests_routing_rules_nm.yml-legacy-ANSIBLE-2-RHEL_6_10_GA/ansible.log

2022-05-27 19:18:46,907 p=14932 u=root n=ansible | /tmp/tmp.oOaqoFSuMq/image.qcow2 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: Warning: Permanently added '[127.0.0.3]:5238' (RSA) to the list of known hosts.\r\nssh_dispatch_run_fatal: Connection to 127.0.0.3 port 5238: error in libcrypto",
    "unreachable": true
}

> I know we have had this problem before which is why we are using  -o PubkeyAcceptedKeyTypes=+ssh-rsa, but maybe something has changed in 9.1?

The ssh-rsa (SHA1 signatures) are no longer supported in RHEL9. The same was for 9.0. See the following KBA for more details and workarounds:

https://access.redhat.com/solutions/6816771

I verified that using 

update-crypto-policies --set DEFAULT:SHA1

on the rhel 9.1 host allows it to ssh into the rhel6 vm

It looks like all VMs have healthy test results. Please review:

[root@netqe26 Upstream-testsuite]# uname -r
5.14.0-105.el9.x86_64
[root@netqe26 Upstream-testsuite]# rpm -q rhel-system-roles
rhel-system-roles-1.18.0-1.el9.noarch

From http://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/rhel-system-roles/1.18.0/1.el9/noarch/rhe\
l-system-roles-1.18.0-1.el9.noarch.rpm

:: [ 14:29:19 ] :: [   PASS   ] :: Test network (/tmp/tmp.iGeKQBkf6A/roles/rhel-system-roles.network/tests/tes\
ts_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_6_10_GA (Expected 0, got 0)
:: [ 14:31:35 ] :: [   PASS   ] :: Test network (/tmp/tmp.NrmQGcqaZh/roles/rhel-system-roles.network/tests/tes\
ts_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_7_9_GA (Expected 0, got 0)
:: [ 14:33:59 ] :: [   PASS   ] :: Test network (/tmp/tmp.o1YvlRF4Qt/roles/rhel-system-roles.network/tests/tes\
ts_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_5_0_GA (Expected 0, got 0)
:: [ 14:45:39 ] :: [   PASS   ] :: Test network (/tmp/tmp.ZEy1VgWazx/roles/rhel-system-roles.network/tests/tes\
ts_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_6_TESTING (Expected 0, got 0)
:: [ 14:49:11 ] :: [   PASS   ] :: Test network (/tmp/tmp.UdoYWuFK1l/roles/rhel-system-roles.network/tests/tes\
ts_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_6_TESTING (Expected 0, got 0)
:: [ 14:51:40 ] :: [   PASS   ] :: Test network (/tmp/tmp.QXHMdZgUY8/roles/rhel-system-roles.network/tests/tes\
ts_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_9_0_TESTING (Expected 0, got 0)
:: [ 14:54:06 ] :: [   PASS   ] :: Test network (/tmp/tmp.nk8miF2FZz/roles/rhel-system-roles.network/tests/tes\
ts_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_9_1_TESTING (Expected 0, got 0)

Triple checked to make sure RHEL_8_7_TESTING case did indeed run correct and it did. I think the duplicate 8.6 was just a manual error on my part from the pre-verify 

[root@netqe6 ~]# uname -r
5.14.0-105.el9.x86_64
[root@netqe6 ~]# rpm -q rhel-system-roles
rhel-system-roles-1.18.0-1.el9.noarch


From http://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/rhel-system-roles/1.18.0/1.el9/noarch/rhel-system-roles-1.18.0-1.el9.noarch.rpm

:: [ 16:15:46 ] :: [   PASS   ] :: Test network (/tmp/tmp.OP23rAMJbl/roles/rhel-system-roles.network/tests/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_7_TESTING (Expected 0, got 0)

I will do the final verify with tomorrows build.

The most recent build is 20220606.1.

The build can't seem to see any of the images again. The following error happened for all target qcow2 images.

:: [ 10:57:58 ] :: [   FAIL   ] :: Downloading image RHEL_9_1_TESTING FAILED. Can not continue testing!

[root@netqe6 Upstream-testsuite]# uname -r
5.14.0-105.el9.x86_64
[root@netqe6 Upstream-testsuite]# rpm -q rhel-system-roles
rhel-system-roles-1.18.0-1.el9.noarch

Can someone take a look at this?

(In reply to Jon Trossbach from comment #22)
> The most recent build is 20220606.1.
> 
> The build can't seem to see any of the images again. The following error
> happened for all target qcow2 images.
> 
> :: [ 10:57:58 ] :: [   FAIL   ] :: Downloading image RHEL_9_1_TESTING
> FAILED. Can not continue testing!
> 
> [root@netqe6 Upstream-testsuite]# uname -r
> 5.14.0-105.el9.x86_64
> [root@netqe6 Upstream-testsuite]# rpm -q rhel-system-roles
> rhel-system-roles-1.18.0-1.el9.noarch
> 
> Can someone take a look at this?

Can you post more of the log?  This could be due to
1) transient network error - I can go to http://download.eng.bos.redhat.com/rhel-9/nightly/RHEL-9/latest-RHEL-9.1/compose/BaseOS/x86_64/images/ in my web browser at the moment - there may be indications earlier in the log that there were network problems at the time
2) python library error - there may be some indication of this earlier in the logs

It appears you may have been right, it is now pulling.

The results are as follows, moving to VERIFIED.

From http://download.eng.bos.redhat.com/brewroot/vol/rhel-9/packages/rhel-system-roles/1.18.0/1.el9/noarch/rhel-system-roles-1.18.0-1.el9.noarch.rpm

[root@netqe16 Upstream-testsuite]# uname -r
5.14.0-105.el9.x86_64
[root@netqe16 Upstream-testsuite]# rpm -q rhel-system-roles
rhel-system-roles-1.18.0-1.el9.noarch

:: [ 13:46:37 ] :: [   PASS   ] :: Test network (/tmp/tmp.ioUbVRnsAt/roles/rhel-system-roles.network/tests/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_6_10_GA (Expected 0, got 0)
:: [ 13:51:53 ] :: [   PASS   ] :: Test network (/tmp/tmp.KPtAGuyRCG/roles/rhel-system-roles.network/tests/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_7_9_GA (Expected 0, got 0)
:: [ 13:55:41 ] :: [   PASS   ] :: Test network (/tmp/tmp.DogHbu5A0O/roles/rhel-system-roles.network/tests/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_5_0_GA (Expected 0, got 0)
:: [ 14:00:02 ] :: [   PASS   ] :: Test network (/tmp/tmp.rUA48RY9j0/roles/rhel-system-roles.network/tests/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_6_TESTING (Expected 0, got 0)
:: [ 14:03:28 ] :: [   PASS   ] :: Test network (/tmp/tmp.jNzoP8qEgH/roles/rhel-system-roles.network/tests/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_8_7_TESTING (Expected 0, got 0)
:: [ 14:05:24 ] :: [   PASS   ] :: Test network (/tmp/tmp.6RMz1MKSmP/roles/rhel-system-roles.network/tests/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_9_0_TESTING (Expected 0, got 0)
:: [ 14:07:10 ] :: [   PASS   ] :: Test network (/tmp/tmp.pHI98RSuWl/roles/rhel-system-roles.network/tests/tests_routing_rules_nm.yml) with ANSIBLE-2.9 against RHEL_9_1_TESTING (Expected 0, got 0)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rhel-system-roles bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:8117