Description of problem: Our Whole disk regression test is failing when running 'disk-image-create ...' with error 'returned non-zero exit status 1 ... unclosed file <_io.BufferedReader name=7>" Version-Release number of selected component (if applicable): 17.1 How reproducible: every time Steps to Reproduce: 1. Run whole disk regression test 2. 3. Actual results: disk-image create fails Expected results: disk-image create succeeds Additional info: Error from log: Using config files: ['/usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-uefi-python3.yaml', '/usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-uefi-rhel9.yaml'] 2022-12-09 16:49:41.544 | imagename: overcloud-hardened-uefi-full 2022-12-09 16:49:41.546 | looking for image at path: ./overcloud-hardened-uefi-full 2022-12-09 16:49:41.547 | Exception occured while running the command 2022-12-09 16:49:41.548 | Traceback (most recent call last): 2022-12-09 16:49:41.550 | File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run 2022-12-09 16:49:41.551 | super(Command, self).run(parsed_args) 2022-12-09 16:49:41.552 | File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run 2022-12-09 16:49:41.554 | return super(Command, self).run(parsed_args) 2022-12-09 16:49:41.555 | File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run 2022-12-09 16:49:41.557 | return_code = self.take_action(parsed_args) or 0 2022-12-09 16:49:41.558 | File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_image.py", line 120, in take_action 2022-12-09 16:49:41.559 | manager.build() 2022-12-09 16:49:41.561 | File "/usr/lib/python3.9/site-packages/tripleo_common/image/build.py", line 84, in build 2022-12-09 16:49:41.562 | builder.build_image(image_path, image_type, node_dist, arch, 2022-12-09 16:49:41.563 | File "/usr/lib/python3.9/site-packages/tripleo_common/image/image_builder.py", line 140, in build_image 2022-12-09 16:49:41.565 | raise subprocess.CalledProcessError(process.returncode, cmd) 2022-12-09 16:49:41.566 | subprocess.CalledProcessError: Command '['disk-image-create', '-a', 'amd64', '-o', './overcloud-hardened-uefi-full', '-t', 'qcow2', '-p', 'python3-psutil,python3-debtcollector,sos,device-mapper-multipath,openstack-heat-agents,os-net-config,jq,rhosp-release', '--min-tmpfs=7', 'rhel', 'openvswitch', 'overcloud-agent', 'overcloud-base', 'overcloud-controller', 'overcloud-compute', 'overcloud-ceph-storage', 'puppet-modules', 'stable-interface-names', 'bootloader', 'element-manifest', 'dynamic-login', 'iptables', 'enable-packages-install', 'override-pip-and-virtualenv', 'dracut-regenerate', 'remove-resolvconf', 'modprobe', 'overcloud-partition-uefi', 'overcloud-secure', 'openssh', 'disable-nouveau', 'reset-bls-entries', 'interface-names']' returned non-zero exit status 1. 2022-12-09 16:49:41.567 | Command '['disk-image-create', '-a', 'amd64', '-o', './overcloud-hardened-uefi-full', '-t', 'qcow2', '-p', 'python3-psutil,python3-debtcollector,sos,device-mapper-multipath,openstack-heat-agents,os-net-config,jq,rhosp-release', '--min-tmpfs=7', 'rhel', 'openvswitch', 'overcloud-agent', 'overcloud-base', 'overcloud-controller', 'overcloud-compute', 'overcloud-ceph-storage', 'puppet-modules', 'stable-interface-names', 'bootloader', 'element-manifest', 'dynamic-login', 'iptables', 'enable-packages-install', 'override-pip-and-virtualenv', 'dracut-regenerate', 'remove-resolvconf', 'modprobe', 'overcloud-partition-uefi', 'overcloud-secure', 'openssh', 'disable-nouveau', 'reset-bls-entries', 'interface-names']' returned non-zero exit status 1. 2022-12-09 16:49:41.569 | /usr/lib/python3.9/site-packages/cliff/app.py:422: ResourceWarning: unclosed file <_io.BufferedReader name=7> 2022-12-09 16:49:41.570 | del err 2022-12-09 16:49:41.571 | ResourceWarning: Enable tracemalloc to get the object allocation traceback 2022-12-09 16:49:41.573 | END return value: 1
Looks odd, one thing I saw in looking at things was weak deps were being installed as well as desktop related bits ... so would want to make sure peek at job/elements to see where those dependencies are creeping in from 2022-12-09 16:49:40.094 | Installing: 2022-12-09 16:49:40.095 | openstack-ironic-python-agent-builder noarch 1:2.8.0-1.20220727182711.e0b51e0.el9ost rhelosp-17.1 48 k 2022-12-09 16:49:40.097 | openstack-tripleo-image-elements noarch 13.1.3-1.20221022031001.cfc336b.el9ost rhelosp-17.1 99 k 2022-12-09 16:49:40.098 | openstack-tripleo-puppet-elements noarch 14.1.3-1.20220727204915.082a9aa.el9ost rhelosp-17.1 36 k ... whole bunch of extra libaries and packages being installed including libX11 related packages, sound and cups 2022-12-09 16:49:40.345 | Installing weak dependencies: 2022-12-09 16:49:40.346 | dconf x86_64 0.40.0-6.el9 rhosp-rhel-9.1-appstream 117 k 2022-12-09 16:49:40.348 | exiv2 x86_64 0.27.5-2.el9 rhosp-rhel-9.1-appstream 984 k 2022-12-09 16:49:40.349 | flatpak x86_64 1.12.7-2.el9 rhosp-rhel-9.1-appstream 1.7 M 2022-12-09 16:49:40.351 | libcanberra-gtk2 x86_64 0.30-26.el9 rhosp-rhel-9.1-appstream 28 k 2022-12-09 16:49:40.352 | p11-kit-server x86_64 0.24.1-2.el9 rhosp-rhel-9.1-appstream 200 k 2022-12-09 16:49:40.354 | pipewire x86_64 0.3.47-2.el9_0 rhosp-rhel-9.1-appstream 41 k 2022-12-09 16:49:40.355 | pipewire-alsa x86_64 0.3.47-2.el9_0 rhosp-rhel-9.1-appstream 61 k 2022-12-09 16:49:40.356 | pipewire-jack-audio-connection-kit x86_64 0.3.47-2.el9_0 rhosp-rhel-9.1-appstream 135 k 2022-12-09 16:49:40.358 | pipewire-pulseaudio x86_64 0.3.47-2.el9_0 rhosp-rhel-9.1-appstream 26 k 2022-12-09 16:49:40.359 | python3-numpy x86_64 1:1.20.1-5.el9 rhosp-rhel-9.1-appstream 5.1 M 2022-12-09 16:49:40.361 | python3-pydot noarch 1.4.1-5.el9ost rhelosp-17.1 48 k 2022-12-09 16:49:40.362 | python3-scipy x86_64 1.6.2-8.el9 rhosp-rhel-9.1-appstream 16 M 2022-12-09 16:49:40.363 | tracker-miners x86_64 3.1.2-1.el9 rhosp-rhel-9.1-appstream 944 k 2022-12-09 16:49:40.365 | xdg-desktop-portal-gtk x86_64 1.12.0-3.el9 rhosp-rhel-9.1-appstream 139 k 2022-12-09 16:49:40.366 | xdg-utils noarch 1.1.3-11.el9 rhosp-rhel-9.1-appstream 78 k Might be as simple as making sure we don't install weak deps in the images The following tweak is used several places 74:echo "install_weak_deps=False" >> /etc/yum.conf
Will investigate not installing soft-dependencies, and finding the package causing this.
The dependency chain is: openstack-ironic-python-agent-builder -> diskimage-builder -> python3-networkx -> the rest of the desktop stack The diskimage-builder block device code uses networkx to build a graph structure, but doesn't require any graphic presentation of that graph. Investigation continues
This change will fix this issue once it is backported upstream and downstream: https://review.opendev.org/c/openstack/python-tripleoclient/+/868147 This change will workaround the issue to allow this job to progress: https://code.engineering.redhat.com/gerrit/c/rhos-qe-core-installer/+/438017
https://review.opendev.org/c/openstack/python-tripleoclient/+/874365 needs to get reviewed and merged asap so we can get it in the release pipeline for 17.1. Tests are failing. Marking BZ as a blocker.
Fix is in compose RHOS-17.1-RHEL-8-20230309.n.1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577