see: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.5-informing#promote-release-openshift-machine-os-content-e2e-aws-4.5 and: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/promote-release-openshift-machine-os-content-e2e-aws-4.5/1341114108363149312 the job is not getting through init: error: could not read dir /usr/local/e2e-aws for template: open /usr/local/e2e-aws: no such file or directory
@skunkerk can you have a look at this job? https://github.com/openshift/release/blob/master/ci-operator/jobs/openshift/release/openshift-release-release-4.5-periodics.yaml#L18 Strangely it doesn't look like it has changed in the last 3 months; you'll probably have to work with the DPTP folks to understand why the directory is not available in the image being used for the job.
Seting to blocker-, this doesn't seem related to an issue that would hold the 4.7 release.
The job is now failing in new and exciting ways Last failure related to `error: could not read dir /usr/local/e2e-aws for template: open /usr/local/e2e-aws: no such file or directory` was at ~15:02 today - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/promote-release-openshift-machine-os-content-e2e-aws-4.5/1347272345747722240 Then started failing like this: ``` E0107 20:18:22.588815 97 config.go:236] while trying to parse blob "{\"auths\":{\"cloud.openshift.com\":{\"auth\":\"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29wZW5zaGlmdGNpdGVzdDFjemttd3R6ZGJlbjdqb21tbzVhcjF5M2FpMTpPQkw4TVVXVUNDV1dUWDQwMEJMNFpGVVM0RkxSOEEzNTJSMjdMQzdIQ1JCRUhVQUVNS0Y3WUZHQTRCODJYNVpF\",\"email\":\"ccoleman+openshift-ci-test\"},\"quay.io\":{\"auth\":\"b3BlbnNoaWZ0LXJlbGVhc2UtZGV2K29wZW5zaGlmdGNpdGVzdDFjemttd3R6ZGJlbjdqb21tbzVhcjF5M2FpMTpPQkw4TVVXVUNDV1dUWDQwMEJMNFpGVVM0RkxSOEEzNTJSMjdMQzdIQ1JCRUhVQUVNS0Y3WUZHQTRCODJYNVpF\",\"email\":\"ccoleman+openshift-ci-test\"},\"registry.ci.openshift.org\":{\"auth\":\"c2VydmljZWFjY291bnQ6ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrRndTekF0YjBaNGJXMUZURXRHTVMwMFVEa3djbEEwUTJWQlRUZERNMGRXUkZwdmJGOVllaTFEUW5NaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpqYVNJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZqY21WMExtNWhiV1VpT2lKcGJXRm5aUzF3ZFd4c1pYSXRkRzlyWlc0dGRqSnRjWE1pTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pYVcxaFoyVXRjSFZzYkdWeUlpd2lhM1ZpWlhKdVpYUmxjeTVwYnk5elpYSjJhV05sWVdOamIzVnVkQzl6WlhKMmFXTmxMV0ZqWTI5MWJuUXVkV2xrSWpvaVlUYzNaamswWVRJdFpUQTNOUzAwTVdJNExUazVaR0V0WkRaa09XUXlZV0kxTVdNNUlpd2ljM1ZpSWpvaWMzbHpkR1Z0T25ObGNuWnBZMlZoWTJOdmRXNTBPbU5wT21sdFlXZGxMWEIxYkd4bGNpSjkuZGx5RjkxREs0V2JLZW1lenFfTDN2bkNwcmVqN2Zkb0ZfQWUxQ2VQMnJFS29Pc0QwYkhsYzA5enp1XzhKSmx0MkdYQmlwSlQ0Qm1GWFlaVjA2RTBTNlY1X2p6cTI3TFhpSXltODR4Yk8yZkg0Q0lhV05NVTFpVWYxc19NTi10b1NHdVFGR0ZyN3QwZGFZSngzcFlIYTVrV0lJcnJJWlVYNVhOY2tDM0thYTFEeGtFanRURUVOaUwtN0xkN1RmOWFlUlAwZDFoc2U1OG1id2o2T1RIU2thRkZfUWMxTTlRNHZKMEdyR2JzRDg3Vnk0Z0dkWjU2NkU5d3NWQ0o4Vk52dFdaWUFkdGN1WjlWOC1faENTbnA3M19LWnFZaGpybVdMYjlVQkJhLVJIOXNiMzFHaGdJMG9Iak5hLUdIZ2djQXVGV0FLcXJsMUhSc1hwMGl2UEJvQ3VBc2VydmljZWFjY291bnQ6ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrRndTekF0YjBaNGJXMUZURXRHTVMwMFVEa3djbEEwUTJWQlRUZERNMGRXUkZwdmJGOVllaTFEUW5NaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpqYVNJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZqY21WMExtNWhiV1VpT2lKcGJXRm5aUzF3ZFd4c1pYSXRkRzlyWlc0dE9HbzJOVGtpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pYVcxaFoyVXRjSFZzYkdWeUlpd2lhM1ZpWlhKdVpYUmxjeTVwYnk5elpYSjJhV05sWVdOamIzVnVkQzl6WlhKMmFXTmxMV0ZqWTI5MWJuUXVkV2xrSWpvaVlUYzNaamswWVRJdFpUQTNOUzAwTVdJNExUazVaR0V0WkRaa09XUXlZV0kxTVdNNUlpd2ljM1ZpSWpvaWMzbHpkR1Z0T25ObGNuWnBZMlZoWTJOdmRXNTBPbU5wT21sdFlXZGxMWEIxYkd4bGNpSjkucGNXSkNCREV2R1hpa3dTMkE3QlJFbmhMeTdxTVkyWjlFTDFIVTZQNDI0RXZoWjhSRzkwcnA2dWpoRUJtQVBaQVdoY2I0VGdjenVsU2RrVlpDUGoxblZZN0tzZVZBd1FTS1QyZkdWTjNEbXBWcHBLOGVmZjFMc1ZjQVp0bWhpWEN0bnNVWnRsTzFQaGt4b0pWWHBSZjIyQV92NEhYSDM5WUZia2stQklJMXJIRG5NWXE1YkR1anVyQzB3T2hYMngwaUFiZGdfUnJBNTloaEZYRTNTMllPUmVYU284MXZhMG9KU29wTzBxY2xzYWwxLXM0VEdIUVJ1QjRiQzJFdUdIVUZFQWpvbi1fVVNBTGk3bU1UQmUteDZXUVZZUUlfbVlydVdRcGVWVF80eEFDeGNPQ0NjdG1FOWRaeElpQXpvTUQ3dUM5QVN2U3NOYk84Q1dUMExIelB3\"},\"registry.connect.redhat.com\":{\"auth\":\"NTE5OTgxNTN8dWhjLTFDektNV1R6RGJlTjdqT01tTzVBcjFZM0FpMTpleUpoYkdjaU9pSlNVelV4TWlKOS5leUp6ZFdJaU9pSXhPVFUzTldSalltRXlOemcwT1dFeU9EVmpNbVl3TXpSbE0yRTBNREkzTVNKOS5NdDJSdEFDY2xrT2FqX2tHa3FRNzlIamNfN1hid0FUYmR5c0wxYUdJajU2anBobUdSQU4yalRNRE5YczVmMVdtQXNOV2hrTThwSGdpVTBNUHpvVjVNaTN3Z0lWVkl0MzJCb28tbGNQUFBSQzc5MVpWdnZyeGkxd3JqdTJwVk5ocHFjRHdYMFVkQTY0ZU95VTZjR2dXRDR3YllKQTJYdnE0MjlULWFud3psamFxMkxRSFp4SFRoMjJ4SnJQUEN6Y3VhSWZDSWZKeUVONVdxanVscXYwb3BtTGhJNVFZeDZEeGlldHp6N3kxWUxweVRiUkF6NFVtbDRjMEp4OUJRNWZWZ0J6WUxqY2s2dXdfdWpUODQ3Q21Fb0xMcm9BV2hDUi0yMUpsY3FKUjVSM2paM0haOEg4SUZxb09rWTBEQXJ5akIzSkhtYXc3UmVlZzhuZnJPMUcyOXZNRFNkVVdleUd5OFMtNmlzWUlBVC1yMm9OdTBDcXdndVB5YTlJMGVmTm1QRXJzYmFCd2FkRGoxMU5abFdha094XzBfYlh3b1V2cWV4Mkg3Ylo4el9pUnhrdE1Qenl0MkdCam8yanp6RUdmVGlHeVhpbUw5d25xQ01HMUtRZm5ZY2RKelE4YUtaTDVwQm4wTkgwRzJUQVNYOFBCWWt3ZnZJc3Bmb2JaR2M2ZGR4S3VLdmFfY08yX1dKN1lISWFwS1dBa3QwWVZWcTdYeEExd2dreE9POUNDd3NHenloLWloYXFsajFJZkFKX2duWVBtRTc1dU4zMl90eVRYYnN6TmxSVTdral9RWlpEZUthdUFQdUVmY0NFLUc4VHVRdUMzd1pOYi1fN1dTQTdwWWdvdGRpcy00eVc3NWpWSEljTlpBbGhVRUcyWGNrcXlEZzZ6NVRzWkJBOA==\",\"email\":\"ccoleman+openshift-ci-test\"},\"registry.redhat.io\":{\"auth\":\"NTE5OTgxNTN8dWhjLTFDektNV1R6RGJlTjdqT01tTzVBcjFZM0FpMTpleUpoYkdjaU9pSlNVelV4TWlKOS5leUp6ZFdJaU9pSXhPVFUzTldSalltRXlOemcwT1dFeU9EVmpNbVl3TXpSbE0yRTBNREkzTVNKOS5NdDJSdEFDY2xrT2FqX2tHa3FRNzlIamNfN1hid0FUYmR5c0wxYUdJajU2anBobUdSQU4yalRNRE5YczVmMVdtQXNOV2hrTThwSGdpVTBNUHpvVjVNaTN3Z0lWVkl0MzJCb28tbGNQUFBSQzc5MVpWdnZyeGkxd3JqdTJwVk5ocHFjRHdYMFVkQTY0ZU95VTZjR2dXRDR3YllKQTJYdnE0MjlULWFud3psamFxMkxRSFp4SFRoMjJ4SnJQUEN6Y3VhSWZDSWZKeUVONVdxanVscXYwb3BtTGhJNVFZeDZEeGlldHp6N3kxWUxweVRiUkF6NFVtbDRjMEp4OUJRNWZWZ0J6WUxqY2s2dXdfdWpUODQ3Q21Fb0xMcm9BV2hDUi0yMUpsY3FKUjVSM2paM0haOEg4SUZxb09rWTBEQXJ5akIzSkhtYXc3UmVlZzhuZnJPMUcyOXZNRFNkVVdleUd5OFMtNmlzWUlBVC1yMm9OdTBDcXdndVB5YTlJMGVmTm1QRXJzYmFCd2FkRGoxMU5abFdha094XzBfYlh3b1V2cWV4Mkg3Ylo4el9pUnhrdE1Qenl0MkdCam8yanp6RUdmVGlHeVhpbUw5d25xQ01HMUtRZm5ZY2RKelE4YUtaTDVwQm4wTkgwRzJUQVNYOFBCWWt3ZnZJc3Bmb2JaR2M2ZGR4S3VLdmFfY08yX1dKN1lISWFwS1dBa3QwWVZWcTdYeEExd2dreE9POUNDd3NHenloLWloYXFsajFJZkFKX2duWVBtRTc1dU4zMl90eVRYYnN6TmxSVTdral9RWlpEZUthdUFQdUVmY0NFLUc4VHVRdUMzd1pOYi1fN1dTQTdwWWdvdGRpcy00eVc3NWpWSEljTlpBbGhVRUcyWGNrcXlEZzZ6NVRzWkJBOA==\",\"email\":\"ccoleman+openshift-ci-test\"},\"registry.svc.ci.openshift.org\":{\"auth\":\"c2VydmljZWFjY291bnQ6ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklpSjkuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObGNuWnBZMlZoWTJOdmRXNTBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5dVlXMWxjM0JoWTJVaU9pSmphU0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVmpjbVYwTG01aGJXVWlPaUpwYldGblpTMXdkV3hzWlhJdGRHOXJaVzR0Y0hOM1oyY2lMQ0pyZFdKbGNtNWxkR1Z6TG1sdkwzTmxjblpwWTJWaFkyTnZkVzUwTDNObGNuWnBZMlV0WVdOamIzVnVkQzV1WVcxbElqb2lhVzFoWjJVdGNIVnNiR1Z5SWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXpaWEoyYVdObExXRmpZMjkxYm5RdWRXbGtJam9pT1RjNE9EazJZVGN0TVRRM01TMHhNV1ZpTFRnMVlXVXROREl3TVRCaE9HVXdNREEwSWl3aWMzVmlJam9pYzNsemRHVnRPbk5sY25acFkyVmhZMk52ZFc1ME9tTnBPbWx0WVdkbExYQjFiR3hsY2lKOS5UWVZKZ0kzRWt4bEFmYnpnb2NpcXFwRUpYQTRBc3BjcU90cGZzUjAyeTZLN0pBa1FmSVhsU2twallUREFDMGFoU3pCN256MExHZmxDcDJUTkFEV19RMTQ3NFF2UWxFYzJMV193blIwMEhVeG9NZGc2bHdSV3dFWkJlelVXaU9Gbm9CVFFTMGFObjNCbThyZ3MwSHNLY0N6ZzRzaXpCUlJlSHVQX3IyVDRybmpFcTBSa090N2ZhQmpZVjFRdkswVDZHZGRKOTNpVThVMmloWEFWNkExWG40R180dC13dmhueW9TNDBlU2VvcTh2Y0g3VGFvYWluSDlZcGRNRnpac3BHZHJsbWF4V1Z0TlVSdzc3N1pwTDctYmFobVNSQVZiQnBWTUVaMzJfQUlFQnlucDBZeXVwWW44amNSazA0Wjlqc2VVZVRydkFRam9sRHNXeWpZSzd3b0E=c2VydmljZWFjY291bnQ6ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklpSjkuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObGNuWnBZMlZoWTJOdmRXNTBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5dVlXMWxjM0JoWTJVaU9pSmphU0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVmpjbVYwTG01aGJXVWlPaUpwYldGblpTMXdkV3hzWlhJdGRHOXJaVzR0Y3pJMk4yc2lMQ0pyZFdKbGNtNWxkR1Z6TG1sdkwzTmxjblpwWTJWaFkyTnZkVzUwTDNObGNuWnBZMlV0WVdOamIzVnVkQzV1WVcxbElqb2lhVzFoWjJVdGNIVnNiR1Z5SWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXpaWEoyYVdObExXRmpZMjkxYm5RdWRXbGtJam9pT1RjNE9EazJZVGN0TVRRM01TMHhNV1ZpTFRnMVlXVXROREl3TVRCaE9HVXdNREEwSWl3aWMzVmlJam9pYzNsemRHVnRPbk5sY25acFkyVmhZMk52ZFc1ME9tTnBPbWx0WVdkbExYQjFiR3hsY2lKOS5mRXpxVE42N3BUTVF3eUdXaGFONkEtRjU3ekV2M2t4WTJZUVVlYXBLX0VQZlQ1c3RhRHZwX0VmWmdkOHRybE5oYl95X1ZjVXhDdHRmbk9sa1ZZRUdKZU13NXFsWlpnX1VDLVNOV1dGaU8xZW00Wmc0T3VoR1U3akljZWt6QmxORGpBTnBIZ3Z4N1VCTDFpY3puZWtNUk9VM1hqa3pERUxsbmlUSVNfc01LeE9PczlzZXBNSjBrR1M4ODVjamJWQW1MTVhiamUxb0dzay1sRGpwbVFrSGZOclpZaU5WdmFyVjdRVFA1M0xmOGtuQXlOdUtTQUp0WWIwMmVtdHdFZEllNkFPb190VFdJVXhudmlUbFUyV2hqN2wtdV90SGN0SmF1cnRCYVFUVFlxUExBekZIb2V0WE15bWUxczNkcU1DZ240TEZ3eUQwMTMzZTZfaXBGOUxrQWc=\"}}}": illegal base64 data at input byte 1152 error: unable to load --registry-config: illegal base64 data at input byte 1152 ``` ...15 minutes later - https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/promote-release-openshift-machine-os-content-e2e-aws-4.5/1347276373848231936 And consistently failing like this. I did a quick comparison of the 4.4/4.5/4.6 job and noticed that this mount definition was missing from the `volumeMount` section of the 4.5 job: ``` - mountPath: /usr/local/e2e-aws name: job-definition subPath: cluster-launch-installer-e2e.yaml ``` 4.4 job def - https://github.com/openshift/release/blob/master/ci-operator/jobs/openshift/release/openshift-release-release-4.4-periodics.yaml#L124-L130 4.5 job def - https://github.com/openshift/release/blob/master/ci-operator/jobs/openshift/release/openshift-release-release-4.5-periodics.yaml#L122-L125 4.6 job def - https://github.com/openshift/release/blob/master/ci-operator/jobs/openshift/release/openshift-release-release-4.6-periodics.yaml#L122-L128 Seems like an easy change to make.
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.5-informing#promote-release-openshift-machine-os-content-e2e-aws-4.5 looks green for the past two days. Closing as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633