Bug 2088583

Summary:	libguestfs: error: download: /boot/loader/entries/ostree-1-rhcos.conf: No such file or directory
Product:	OpenShift Container Platform	Reporter:	Petr Antonov <antpeter>
Component:	Bare Metal Hardware Provisioning	Assignee:	Tomas Sedovic <tsedovic>
Bare Metal Hardware Provisioning sub component:	ironic	QA Contact:	Jad Haj Yahya <jhajyahy>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	antpeter, jhajyahy, lshilin, rpittau, vrutkovs, zbitter
Version:	4.11	Keywords:	Triaged
Target Milestone:	---
Target Release:	4.12.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-01-17 19:48:59 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2111636

Description Petr Antonov 2022-05-19 18:09:42 UTC

Description of problem:

There were an error during OKD baremetal IPI install: metal3 deployment in openshift-machine-api namespace could not be started properly for some reason

Version-Release number of selected component (if applicable):

OKD Version: 4.10.0-0.okd-2022-05-07-021833
How reproducible:
100%

Steps to Reproduce:
```
export OCP_RELEASE=4.10.0-0.okd-2022-05-07-021833

wget -c https://github.com/openshift/okd/releases/download/${OCP_RELEASE}/openshift-client-linux-${OCP_RELEASE}.tar.gz -O - | tar -xz -C /usr/bin/

export RELEASE_IMAGE=$(curl -sL https://github.com/openshift/okd/releases/download/${OCP_RELEASE}/release.txt | grep 'Pull From: quay.io' | awk -F ' ' '{print $3}')
export cmd=openshift-baremetal-install
export extract_dir=$(pwd)
oc adm release extract --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}
mv openshift-baremetal-install ~/bin
openshift-baremetal-install version
``
install-config.yaml:

```
apiVersion: v1
baseDomain: cloud.local
metadata:
  name: ocp01
networking:
  machineCIDR: xxx.0/24
  networkType: OVNKubernetes
  clusterNetwork:
  - cidr: 10.13.0.0/16
    hostPrefix: 23
  serviceNetwork:
  - 10.14.0.0/16
compute:
- name: worker
  replicas: 0 
controlPlane:
  name: master
  replicas: 3
  platform:
    baremetal: {}
platform:
  baremetal:
    bootstrapOSImage: http://xxx.3:8080/fedora-coreos-35.20220327.3.0-qemu.x86_64.qcow2.xz?sha256=935ef3fc65b79344ebc006e89f45f67606afa5d39f2222377bc3dba89f6adb7d
    clusterOSImage: http://xxx.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz?sha256=5e3e40723288aa56735caa5e5fcb58079da5b27df392ee185a09f9b6742fa93f
    apiVIP: xxx.8
    ingressVIP: xxx.9
    externalBridge: br1.130
    provisioningNetwork: Disabled
    hosts:
      - name: mst01
        role: master
        bmc:
          address: redfish-virtualmedia://xxx1/redfish/v1/Systems/1
          username: xxx
          password: xxx
          disableCertificateVerification: True
        bootMACAddress: xxx
        rootDeviceHints:
         deviceName: "/dev/sdb"
      - name: mst02
        role: master
        bmc:
          address: redfish-virtualmedia://xxx2/redfish/v1/Systems/1
          username: xxx
          password: xxx
          disableCertificateVerification: True
        bootMACAddress: xxx
        rootDeviceHints:
         deviceName: "/dev/sdb"
      - name: mst03
        role: master
        bmc:
          address: redfish-virtualmedia://xxx3/redfish/v1/Systems/1
          username: xxx
          password: xxx
          disableCertificateVerification: True
        bootMACAddress: xxx
        rootDeviceHints:
         deviceName: "/dev/sdb"

pullSecret: ''
sshKey: 'ssh-ed25519 AAAA root'
```

Please see the log output of metal3-machine-os-downloader init-container in metal3 pod:

```
+ export http_proxy=
+ http_proxy=
+ export https_proxy=
+ https_proxy=
+ export no_proxy=
+ no_proxy=
+ export CURL_CA_BUNDLE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
+ CURL_CA_BUNDLE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
+ export IP_OPTIONS=ip=dhcp
+ IP_OPTIONS=ip=dhcp
+ export 'RHCOS_IMAGE_URL=http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz?sha256=5e3e40723288aa56735caa5e5fcb58079da5b27df392ee185a09f9b6742fa93f'
+ RHCOS_IMAGE_URL='http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz?sha256=5e3e40723288aa56735caa5e5fcb58079da5b27df392ee185a09f9b6742fa93f'
+ '[' -z 'http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz?sha256=5e3e40723288aa56735caa5e5fcb58079da5b27df392ee185a09f9b6742fa93f' ']'
++ echo 'http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz?sha256=5e3e40723288aa56735caa5e5fcb58079da5b27df392ee185a09f9b6742fa93f'
++ cut -f 1 -d '?'
+ RHCOS_IMAGE_URL_STRIPPED=http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz
+ [[ http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz =~ qcow2(\.[gx]z)?$ ]]
++ basename http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz
+ RHCOS_IMAGE_FILENAME_RAW=fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz
+ RHCOS_IMAGE_FILENAME_QCOW=fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2
+ IMAGE_FILENAME_EXTENSION=.xz
++ dirname http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz
+ IMAGE_URL=http://10.xxx.ccc.3:8080
+ RHCOS_IMAGE_FILENAME_COMPRESSED=fedora-coreos-35.20220327.3.0-compressed.x86_64.qcow2
+ RHCOS_IMAGE_FILENAME_CACHED=cached-fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2
+ FFILENAME=rhcos-ootpa-latest.qcow2
+ mkdir -p /shared/html/images
+ [[ fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2 == *\-\o\p\e\n\s\t\a\c\k* ]]
+ [[ -s /shared/html/images/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2/fedora-coreos-35.20220327.3.0-compressed.x86_64.qcow2.md5sum ]]
+ mkdir -p /shared/tmp
++ mktemp -d -p /shared/tmp
+ TMPDIR=/shared/tmp/tmp.kZQHYAz4Js
+ trap 'rm -fr /shared/tmp/tmp.kZQHYAz4Js' EXIT
+ cd /shared/tmp/tmp.kZQHYAz4Js
+ clearproxy http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz
+ unset HTTP_PROXY http_proxy HTTPS_PROXY https_proxy
+ '[' -s /shared/html/images/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2/cached-fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.md5sum ']'
+ CONNECT_TIMEOUT=120
+ MAX_ATTEMPTS=5
++ seq 5
+ for i in $(seq ${MAX_ATTEMPTS})
+ curl -v -g --compressed -L --fail --connect-timeout 120 -o fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz http://10.xxx.ccc.3:8080/fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz
*   Trying 10.xxx.ccc.3...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.xxx.ccc.3 (10.xxx.ccc.3) port 8080 (#0)
> GET /fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz HTTP/1.1
> Host: 10.xxx.ccc.3:8080
> User-Agent: curl/7.61.1
> Accept: */*
> Accept-Encoding: deflate, gzip, br
> 
< HTTP/1.1 200 OK
< Date: Mon, 16 May 2022 12:02:41 GMT
< Server: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips
< Last-Modified: Thu, 12 May 2022 08:33:51 GMT
< ETag: "29280ec0-5decc6adfbbd4"
< Accept-Ranges: bytes
< Content-Length: 690491072
< Content-Type: application/x-xz
< 
{ [2628 bytes data]

 90  658M   90  594M    0     0  1038M      0 --:--:-- --:--:-- --:--:-- 1036M
100  658M  100  658M    0     0  1033M      0 --:--:-- --:--:-- --:--:-- 1032M
* Connection #0 to host 10.xxx.ccc.3 left intact
+ break
+ [[ .xz == .gz ]]
+ [[ .xz == .xz ]]
+ unxz fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2.xz
+ '[' -n ip=dhcp ']'
++ LIBGUESTFS_BACKEND=direct
++ virt-filesystems -a fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2 -l
++ grep boot
++ cut -f1 '-d '
+ BOOT_DISK=/dev/sda3
+ LIBGUESTFS_BACKEND=direct
+ virt-edit -a fedora-coreos-35.20220327.3.0-openstack.x86_64.qcow2 -m /dev/sda3 /boot/loader/entries/ostree-1-rhcos.conf -e 's/^options/options ip=dhcp/'
libguestfs: error: download: /boot/loader/entries/ostree-1-rhcos.conf: No such file or directory
+ rm -fr /shared/tmp/tmp.kZQHYAz4Js
```


Fedora CoreOS image does not have /boot/loader/entries/ostree-1-rhcos.conf file

Actual results:


Expected results:


Additional info:

Comment 1 Zane Bitter 2022-07-25 20:04:41 UTC

clusterOSImage is not used in 4.10 and later, so if you just omit it from the install config then you won't hit this problem.

Comment 2 Riccardo Pittau 2022-07-26 08:34:39 UTC

Petr please verify what Zane suggested when you have a moment, thanks

Comment 4 Jad Haj Yahya 2022-08-11 09:58:39 UTC

ran a 4.12 deployment with bootstrapOSImage but without clusterOSImage and it finished successfully

Comment 7 errata-xmlrpc 2023-01-17 19:48:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399

Comment 8 Red Hat Bugzilla 2023-09-18 04:37:28 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days