Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1926558

Summary: [release-4.4] OCP installs fail on ppc64le/s390x in the bootstrap phase
Product: OpenShift Container Platform Reporter: Prashanth Sundararaman <psundara>
Component: Multi-ArchAssignee: Prashanth Sundararaman <psundara>
Status: CLOSED EOL QA Contact: Barry Donahue <bdonahue>
Severity: high Docs Contact:
Priority: high    
Version: 4.4CC: jpoulin, vlaad
Target Milestone: ---   
Target Release: 4.4.z   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1926564 (view as bug list) Environment:
Last Closed: 2021-03-09 13:54:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1926564    
Bug Blocks:    
Attachments:
Description Flags
package diff for s390x
none
package diff for ppc64le none

Description Prashanth Sundararaman 2021-02-09 03:14:21 UTC 
Description of problem:
Recently discovered in the CI that 4.4 builds are failing in the bootstrap phase.
On manually trying an install , it looks like the error is the kubelet service not starting and this was the error log:

Feb 08 22:02:17 psundara-ocp-ptj44-bootstrap hyperkube[2888]: I0208 22:02:17.946044    2888 server.go:540] No cloud provider specified: "" from the config file: ""
Feb 08 22:02:17 psundara-ocp-ptj44-bootstrap hyperkube[2888]: W0208 22:02:17.946082    2888 server.go:563] standalone mode, no API client
Feb 08 22:02:17 psundara-ocp-ptj44-bootstrap hyperkube[2888]: F0208 22:02:17.946123    2888 server.go:273] failed to run Kubelet: No authentication method configured


There seems to have been a recent change in the installer code that seemed to have indirectly caused this issue: https://github.com/openshift/installer/commit/2e69b437cf204b0f01e0ba1d99097200b75ccb44. 

Looks like the hyperkube package for ppc64le/s390x was old because the bootimage was old.

This change works fine for x86 because the x86 rhcos bootimages were bumped for a CVE: https://github.com/openshift/installer/pull/3985. At that time the ppc64le/s390x images were never bumped.


The bootimages need to be bumped for ppc64le/s390x to fix this issue
Comment 1 Prashanth Sundararaman 2021-02-09 03:16:45 UTC 
Going to update the bootimages to : 

https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.4-s390x/44.82.202011122341-0/s390x/meta.json for s390x
https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.4-ppc64le/44.82.202012010439-0/ppc64le/meta.json for ppc6l4e

these are the closest builds to x86 i could find and this is the hyperkube package diff:

  "openshift-hyperkube": {
            "rhcos-4.4-s390x/44.81.202004252140-0": "openshift-hyperkube-4.4.0-202004241258.git.1.bfb96d1.el8.s390x",
            "rhcos-4.4-s390x/44.82.202011122341-0": "openshift-hyperkube-4.4.0-202011122017.p0.git.0.149ca32.el8.s390x"
        },


 "openshift-hyperkube": {
            "rhcos-4.4-ppc64le/44.81.202004252240-0": "openshift-hyperkube-4.4.0-202004241258.git.1.bfb96d1.el8.ppc64le",
            "rhcos-4.4-ppc64le/44.82.202012010439-0": "openshift-hyperkube-4.4.0-202011130111.p0.git.0.4861dfa.el8.ppc64le"
        },
Comment 2 Prashanth Sundararaman 2021-02-09 03:34:26 UTC 
Created attachment 1755828 [details]
package diff for s390x
Comment 3 Prashanth Sundararaman 2021-02-09 03:34:56 UTC 
Created attachment 1755829 [details]
package diff for ppc64le
Comment 7 Prashanth Sundararaman 2021-02-12 14:10:05 UTC 
Ci jobs for 4.4 are passing again.