Bug 2152197

Summary:	Disk image create failing returning non zero exit status 1 unclosed file
Product:	Red Hat OpenStack	Reporter:	Jason Paroly <jparoly>
Component:	python-tripleoclient	Assignee:	Steve Baker <sbaker>
Status:	CLOSED ERRATA	QA Contact:	David Rosenfeld <drosenfe>
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	17.1 (Wallaby)	CC:	amoralej, apevec, hbrock, jkreger, jschluet, jslagle, mburns, prgutier, pweeks, sbaker
Target Milestone:	rc	Keywords:	Triaged
Target Release:	17.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	python-tripleoclient-16.5.1-1.20230505010953.534fe49.el9ost	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-16 01:13:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jason Paroly 2022-12-09 17:41:06 UTC

Description of problem:
Our Whole disk regression test is failing when running 'disk-image-create ...' with error 'returned non-zero exit status 1 ... unclosed file <_io.BufferedReader name=7>"

Version-Release number of selected component (if applicable):
17.1

How reproducible:
every time

Steps to Reproduce:
1. Run whole disk regression test
2.
3.

Actual results:
disk-image create fails

Expected results:
disk-image create succeeds

Additional info:

Error from log:

Using config files: ['/usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-uefi-python3.yaml', '/usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-uefi-rhel9.yaml']
2022-12-09 16:49:41.544 | imagename: overcloud-hardened-uefi-full
2022-12-09 16:49:41.546 | looking for image at path: ./overcloud-hardened-uefi-full
2022-12-09 16:49:41.547 | Exception occured while running the command
2022-12-09 16:49:41.548 | Traceback (most recent call last):
2022-12-09 16:49:41.550 |   File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run
2022-12-09 16:49:41.551 |     super(Command, self).run(parsed_args)
2022-12-09 16:49:41.552 |   File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2022-12-09 16:49:41.554 |     return super(Command, self).run(parsed_args)
2022-12-09 16:49:41.555 |   File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
2022-12-09 16:49:41.557 |     return_code = self.take_action(parsed_args) or 0
2022-12-09 16:49:41.558 |   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_image.py", line 120, in take_action
2022-12-09 16:49:41.559 |     manager.build()
2022-12-09 16:49:41.561 |   File "/usr/lib/python3.9/site-packages/tripleo_common/image/build.py", line 84, in build
2022-12-09 16:49:41.562 |     builder.build_image(image_path, image_type, node_dist, arch,
2022-12-09 16:49:41.563 |   File "/usr/lib/python3.9/site-packages/tripleo_common/image/image_builder.py", line 140, in build_image
2022-12-09 16:49:41.565 |     raise subprocess.CalledProcessError(process.returncode, cmd)
2022-12-09 16:49:41.566 | subprocess.CalledProcessError: Command '['disk-image-create', '-a', 'amd64', '-o', './overcloud-hardened-uefi-full', '-t', 'qcow2', '-p', 'python3-psutil,python3-debtcollector,sos,device-mapper-multipath,openstack-heat-agents,os-net-config,jq,rhosp-release', '--min-tmpfs=7', 'rhel', 'openvswitch', 'overcloud-agent', 'overcloud-base', 'overcloud-controller', 'overcloud-compute', 'overcloud-ceph-storage', 'puppet-modules', 'stable-interface-names', 'bootloader', 'element-manifest', 'dynamic-login', 'iptables', 'enable-packages-install', 'override-pip-and-virtualenv', 'dracut-regenerate', 'remove-resolvconf', 'modprobe', 'overcloud-partition-uefi', 'overcloud-secure', 'openssh', 'disable-nouveau', 'reset-bls-entries', 'interface-names']' returned non-zero exit status 1.
2022-12-09 16:49:41.567 | Command '['disk-image-create', '-a', 'amd64', '-o', './overcloud-hardened-uefi-full', '-t', 'qcow2', '-p', 'python3-psutil,python3-debtcollector,sos,device-mapper-multipath,openstack-heat-agents,os-net-config,jq,rhosp-release', '--min-tmpfs=7', 'rhel', 'openvswitch', 'overcloud-agent', 'overcloud-base', 'overcloud-controller', 'overcloud-compute', 'overcloud-ceph-storage', 'puppet-modules', 'stable-interface-names', 'bootloader', 'element-manifest', 'dynamic-login', 'iptables', 'enable-packages-install', 'override-pip-and-virtualenv', 'dracut-regenerate', 'remove-resolvconf', 'modprobe', 'overcloud-partition-uefi', 'overcloud-secure', 'openssh', 'disable-nouveau', 'reset-bls-entries', 'interface-names']' returned non-zero exit status 1.
2022-12-09 16:49:41.569 | /usr/lib/python3.9/site-packages/cliff/app.py:422: ResourceWarning: unclosed file <_io.BufferedReader name=7>
2022-12-09 16:49:41.570 |   del err
2022-12-09 16:49:41.571 | ResourceWarning: Enable tracemalloc to get the object allocation traceback
2022-12-09 16:49:41.573 | END return value: 1

Comment 3 Jon Schlueter 2022-12-14 20:23:55 UTC

Looks odd, one thing I saw in looking at things was weak deps were being installed as well as desktop related bits ... so would want to make sure peek at job/elements to see where those dependencies are creeping in from

2022-12-09 16:49:40.094 | Installing:
2022-12-09 16:49:40.095 |  openstack-ironic-python-agent-builder   noarch  1:2.8.0-1.20220727182711.e0b51e0.el9ost rhelosp-17.1               48 k
2022-12-09 16:49:40.097 |  openstack-tripleo-image-elements        noarch  13.1.3-1.20221022031001.cfc336b.el9ost  rhelosp-17.1               99 k
2022-12-09 16:49:40.098 |  openstack-tripleo-puppet-elements       noarch  14.1.3-1.20220727204915.082a9aa.el9ost  rhelosp-17.1               36 k

... whole bunch of extra libaries and packages being installed including libX11 related packages, sound and cups

2022-12-09 16:49:40.345 | Installing weak dependencies:
2022-12-09 16:49:40.346 |  dconf                                   x86_64  0.40.0-6.el9                            rhosp-rhel-9.1-appstream  117 k
2022-12-09 16:49:40.348 |  exiv2                                   x86_64  0.27.5-2.el9                            rhosp-rhel-9.1-appstream  984 k
2022-12-09 16:49:40.349 |  flatpak                                 x86_64  1.12.7-2.el9                            rhosp-rhel-9.1-appstream  1.7 M
2022-12-09 16:49:40.351 |  libcanberra-gtk2                        x86_64  0.30-26.el9                             rhosp-rhel-9.1-appstream   28 k
2022-12-09 16:49:40.352 |  p11-kit-server                          x86_64  0.24.1-2.el9                            rhosp-rhel-9.1-appstream  200 k
2022-12-09 16:49:40.354 |  pipewire                                x86_64  0.3.47-2.el9_0                          rhosp-rhel-9.1-appstream   41 k
2022-12-09 16:49:40.355 |  pipewire-alsa                           x86_64  0.3.47-2.el9_0                          rhosp-rhel-9.1-appstream   61 k
2022-12-09 16:49:40.356 |  pipewire-jack-audio-connection-kit      x86_64  0.3.47-2.el9_0                          rhosp-rhel-9.1-appstream  135 k
2022-12-09 16:49:40.358 |  pipewire-pulseaudio                     x86_64  0.3.47-2.el9_0                          rhosp-rhel-9.1-appstream   26 k
2022-12-09 16:49:40.359 |  python3-numpy                           x86_64  1:1.20.1-5.el9                          rhosp-rhel-9.1-appstream  5.1 M
2022-12-09 16:49:40.361 |  python3-pydot                           noarch  1.4.1-5.el9ost                          rhelosp-17.1               48 k
2022-12-09 16:49:40.362 |  python3-scipy                           x86_64  1.6.2-8.el9                             rhosp-rhel-9.1-appstream   16 M
2022-12-09 16:49:40.363 |  tracker-miners                          x86_64  3.1.2-1.el9                             rhosp-rhel-9.1-appstream  944 k
2022-12-09 16:49:40.365 |  xdg-desktop-portal-gtk                  x86_64  1.12.0-3.el9                            rhosp-rhel-9.1-appstream  139 k
2022-12-09 16:49:40.366 |  xdg-utils                               noarch  1.1.3-11.el9                            rhosp-rhel-9.1-appstream   78 k


Might be as simple as making sure we don't install weak deps in the images

The following tweak is used several places

74:echo "install_weak_deps=False" >> /etc/yum.conf

Comment 4 Steve Baker 2022-12-19 20:43:15 UTC

Will investigate not installing soft-dependencies, and finding the package causing this.

Comment 5 Steve Baker 2022-12-19 22:32:59 UTC

The dependency chain is:

openstack-ironic-python-agent-builder
-> diskimage-builder
   -> python3-networkx
      -> the rest of the desktop stack

The diskimage-builder block device code uses networkx to build a graph structure, but doesn't require any graphic presentation of that graph.

Investigation continues

Comment 6 Steve Baker 2022-12-20 01:35:28 UTC

This change will fix this issue once it is backported upstream and downstream:
https://review.opendev.org/c/openstack/python-tripleoclient/+/868147

This change will workaround the issue to allow this job to progress:
https://code.engineering.redhat.com/gerrit/c/rhos-qe-core-installer/+/438017

Comment 8 Jason Paroly 2023-02-27 20:58:24 UTC

https://review.opendev.org/c/openstack/python-tripleoclient/+/874365 needs to get reviewed and merged asap so we can get it in the release pipeline for 17.1.  Tests are failing.  Marking BZ as a blocker.

Comment 11 Steve Baker 2023-03-14 19:38:09 UTC

Fix is in compose RHOS-17.1-RHEL-8-20230309.n.1

Comment 27 errata-xmlrpc 2023-08-16 01:13:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:4577