Bug 1977473

Summary: Assisted service on IPv6 cluster installed with proxy: the assisted-service container doesn't start " network is unreachable"
Product: OpenShift Container Platform Reporter: Alexander Chuzhoy <sasha>
Component: assisted-installerAssignee: Antoni Segura Puimedon <asegurap>
assisted-installer sub component: Deployment Operator QA Contact: bjacot
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: unspecified CC: aos-bugs, mfilanov, ohochman
Version: 4.8Keywords: Triaged
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AI-Team-Platform
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-12 06:01:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2021-06-29 20:29:48 UTC 
Version:
4.8.0-rc.1
quay.io/acm-d/acm-custom-registry:2.3.0-DOWNSTREAM-2021-06-28-15-34-59

Steps to reproduce:
1. Deploy ipv6 cluster via assisted-installer UI using proxy.
2. Attempt to deploy assisted-installer:


    apiVersion: agent-install.openshift.io/v1beta1
    kind: AgentServiceConfig
    metadata:
      namespace: rhacm
      name: agent
    spec:
      mirrorRegistryRef:
        name: mirror-registry-ca
      osImages:
        - openshiftVersion: "4.8"
          version: "48.84.202106231817-0"
          url: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.1/rhcos-live.x86_64.iso"
          rootFSUrl: "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.1/rhcos-live-rootfs.x86_64.img"
      databaseStorage:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
      filesystemStorage:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi


Result:

oc get pod|grep assi
assisted-service-5cbcd4d878-5jxvg                                 1/2     CrashLoopBackOff   6          9m56s


oc logs assisted-service-5cbcd4d878-5jxvg -c assisted-service 
false
time="2021-06-29T20:27:11Z" level=info msg="Setting log format: text" func=main.InitLogs file="/remote-source/app/cmd/main.go:149"
time="2021-06-29T20:27:11Z" level=info msg="Setting Log Level: info" func=main.InitLogs file="/remote-source/app/cmd/main.go:155"
time="2021-06-29T20:27:11Z" level=info msg="Starting bm service" func=main.main file="/remote-source/app/cmd/main.go:183"
time="2021-06-29T20:27:11Z" level=info msg="Started service with OCP versions {\"4.8\":{\"display_name\":\"4.8\",\"release_image\":null,\"release_version\":null,\"rhcos_image\":\"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.1/rhcos-live.x86_64.iso\",\"rhcos_rootfs\":\"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.1/rhcos-live-rootfs.x86_64.img\",\"rhcos_version\":\"48.84.202106231817-0\",\"support_level\":null}}" func=main.main file="/remote-source/app/cmd/main.go:194"
time="2021-06-29T20:27:11Z" level=info msg="Connecting to DB" func=main.setupDB file="/remote-source/app/cmd/main.go:546"
time="2021-06-29T20:27:11Z" level=info msg="Connected to DB" func=main.setupDB file="/remote-source/app/cmd/main.go:561"
I0629 20:27:12.424379       1 request.go:655] Throttling request took 1.041712728s, request: GET:https://[fd02::1]:443/apis/monitoring.coreos.com/v1?timeout=32s
time="2021-06-29T20:27:16Z" level=info msg="Created local authenticator" func=github.com/openshift/assisted-service/pkg/auth.NewAuthenticator file="/remote-source/app/pkg/auth/authenticator.go:52" pkg=auth
time="2021-06-29T20:27:16Z" level=info msg="Configuring CNV Operator plugin" func=github.com/openshift/assisted-service/internal/operators/cnv.NewCNVOperator file="/remote-source/app/internal/operators/cnv/cnv_operator.go:43" config="{map[10de:1db6:true 10de:1eb8:true] map[15b3:1013:true 15b3:1015:true 15b3:1017:true 15b3:101b:true 8086:158b:true] true}"
time="2021-06-29T20:27:16Z" level=info msg="DeployTarget: k8s" func=main.main file="/remote-source/app/cmd/main.go:267"
W0629 20:27:16.027214       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-06-29T20:27:16Z" level=info msg="Attempting to acquire leader lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection" file="/remote-source/app/pkg/leader/leaderelector.go:120" configMap=assisted-service-leader-election-helper pkg=monitor-runner
W0629 20:27:16.033826       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-06-29T20:27:16Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-leader-election-helper pkg=monitor-runner
I0629 20:27:16.033952       1 leaderelection.go:243] attempting to acquire leader lease rhacm/assisted-service-leader-election-helper...
time="2021-06-29T20:27:16Z" level=info msg="Attempting to acquire leader lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection" file="/remote-source/app/pkg/leader/leaderelector.go:120" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-06-29T20:27:16Z" level=info msg="Start waiting for leader" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).waitForLeader" file="/remote-source/app/pkg/leader/leaderelector.go:87" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-06-29T20:27:16Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-migration-helper pkg=migrationLeader
I0629 20:27:16.037945       1 leaderelection.go:243] attempting to acquire leader lease rhacm/assisted-service-migration-helper...
I0629 20:27:23.130345       1 leaderelection.go:253] successfully acquired lease rhacm/assisted-service-migration-helper
time="2021-06-29T20:27:23Z" level=info msg="Successfully acquired leadership lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func1" file="/remote-source/app/pkg/leader/leaderelector.go:176" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-06-29T20:27:24Z" level=info msg="Got leader, stop waiting" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).waitForLeader" file="/remote-source/app/pkg/leader/leaderelector.go:94" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-06-29T20:27:24Z" level=info msg="Start automigration" func=main.autoMigrationWithLeader.func1 file="/remote-source/app/cmd/main.go:663"
time="2021-06-29T20:27:24Z" level=info msg="Finished automigration" func=main.autoMigrationWithLeader.func1 file="/remote-source/app/cmd/main.go:669"
time="2021-06-29T20:27:24Z" level=info msg="Starting manual migrations" func=main.autoMigrationWithLeader.func1 file="/remote-source/app/cmd/main.go:671"
time="2021-06-29T20:27:24Z" level=info msg="Finished manual migrations" func=main.autoMigrationWithLeader.func1 file="/remote-source/app/cmd/main.go:677"
time="2021-06-29T20:27:24Z" level=info msg="NO LONGER LEADER" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func2" file="/remote-source/app/pkg/leader/leaderelector.go:180" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-06-29T20:27:24Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-06-29T20:27:24Z" level=info msg="Given context was cancelled, exiting leader elector" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/app/pkg/leader/leaderelector.go:128" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-06-29T20:27:24Z" level=info msg="Started Cluster State Monitor" func="github.com/openshift/assisted-service/pkg/thread.(*Thread).Start" file="/remote-source/app/pkg/thread/thread.go:40" pkg=cluster-monitor
time="2021-06-29T20:27:24Z" level=info msg="Started Host State Monitor" func="github.com/openshift/assisted-service/pkg/thread.(*Thread).Start" file="/remote-source/app/pkg/thread/thread.go:40" pkg=host-monitor
time="2021-06-29T20:27:24Z" level=info msg="Started Deletion Worker" func="github.com/openshift/assisted-service/pkg/thread.(*Thread).Start" file="/remote-source/app/pkg/thread/thread.go:40" garbagecollector="Deletion Worker"
time="2021-06-29T20:27:24Z" level=info msg="Started Image Expiration Monitor" func="github.com/openshift/assisted-service/pkg/thread.(*Thread).Start" file="/remote-source/app/pkg/thread/thread.go:40" pkg=image-expiration-monitor
time="2021-06-29T20:27:24Z" level=info msg="waiting for REST api readiness before starting controllers" func=main.main.func4 file="/remote-source/app/cmd/main.go:482"
time="2021-06-29T20:27:24Z" level=info msg="Attempting to acquire leader lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection" file="/remote-source/app/pkg/leader/leaderelector.go:120" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-06-29T20:27:24Z" level=info msg="Start waiting for leader" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).waitForLeader" file="/remote-source/app/pkg/leader/leaderelector.go:87" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-06-29T20:27:24Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
I0629 20:27:24.383138       1 leaderelection.go:243] attempting to acquire leader lease rhacm/assisted-service-baseiso-helper...
I0629 20:27:30.435661       1 leaderelection.go:253] successfully acquired lease rhacm/assisted-service-baseiso-helper
time="2021-06-29T20:27:30Z" level=info msg="Successfully acquired leadership lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func1" file="/remote-source/app/pkg/leader/leaderelector.go:176" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
I0629 20:27:31.125569       1 leaderelection.go:253] successfully acquired lease rhacm/assisted-service-leader-election-helper
time="2021-06-29T20:27:31Z" level=info msg="Successfully acquired leadership lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func1" file="/remote-source/app/pkg/leader/leaderelector.go:176" configMap=assisted-service-leader-election-helper pkg=monitor-runner
time="2021-06-29T20:27:32Z" level=info msg="Got leader, stop waiting" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).waitForLeader" file="/remote-source/app/pkg/leader/leaderelector.go:94" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-06-29T20:27:32Z" level=warning msg="ISO template version metadata not found, uploading latest" func=github.com/openshift/assisted-service/pkg/s3wrapper.HaveLatestMinimalTemplate file="/remote-source/app/pkg/s3wrapper/util.go:120" error="object minimal_templates_version.json was not found"
time="2021-06-29T20:27:32Z" level=info msg="Filesystem '/data/' usage is 1.3%" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClientDecorator).conditionalLog" file="/remote-source/app/pkg/s3wrapper/filesystem.go:426"
time="2021-06-29T20:27:32Z" level=fatal msg="Failed to upload boot files" func=main.main.func1 file="/remote-source/app/cmd/main.go:176" error="Failed uploading boot files for OCP version 4.8: Failed fetching from URL https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.1/rhcos-live.x86_64.iso: Get \"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.1/rhcos-live.x86_64.iso\": dial tcp 54.172.173.155:443: connect: network is unreachable"
time="2021-06-29T20:27:32Z" level=info msg="NO LONGER LEADER" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func2" file="/remote-source/app/pkg/leader/leaderelector.go:180" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-06-29T20:27:32Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
Comment 1 Alexander Chuzhoy 2021-06-29 20:32:07 UTC 
expected result: since the cluster is deployed via proxy this should work and we shouldn't see:

error="Failed uploading boot files for OCP version 4.8: Failed fetching from URL https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.1/rhcos-live.x86_64.iso: Get \"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.1/rhcos-live.x86_64.iso\": dial tcp 54.172.173.155:443: connect: network is unreachable"
Comment 3 Alexander Chuzhoy 2021-06-29 20:59:00 UTC 
As a workaround placed the images on a local web server and was able to start the assisted-service pod/container.

oc get agentserviceconfigs.agent-install.openshift.io agent -o yaml
apiVersion: agent-install.openshift.io/v1beta1
kind: AgentServiceConfig
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"agent-install.openshift.io/v1beta1","kind":"AgentServiceConfig","metadata":{"annotations":{},"name":"agent"},"spec":{"databaseStorage":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}}},"filesystemStorage":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"20Gi"}}},"mirrorRegistryRef":{"name":"mirror-registry-ca"},"osImages":[{"openshiftVersion":"4.8","rootFSUrl":"http://e24-h01-000-r640.rdu2.scalelab.redhat.com/ocp_images/rhcos-live-rootfs.x86_64.img","url":"http://e24-h01-000-r640.rdu2.scalelab.redhat.com/ocp_images/rhcos-live.x86_64.iso","version":"48.84.202106231817-0"}]}}
  creationTimestamp: "2021-06-29T20:11:43Z"
  generation: 5
  name: agent
  resourceVersion: "134120"
  uid: 81c3fc51-67de-4efe-b4f4-09ffcc321c5d
spec:
  databaseStorage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
  filesystemStorage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
  mirrorRegistryRef:
    name: mirror-registry-ca
  osImages:
  - openshiftVersion: "4.8"
    rootFSUrl: http://e24-h01-000-r640.rdu2.scalelab.redhat.com/ocp_images/rhcos-live-rootfs.x86_64.img
    url: http://e24-h01-000-r640.rdu2.scalelab.redhat.com/ocp_images/rhcos-live.x86_64.iso
    version: 48.84.202106231817-0
status:
  conditions:
  - lastTransitionTime: "2021-06-29T20:53:50Z"
    message: AgentServiceConfig reconcile completed without error.
    reason: ReconcileSucceeded
    status: "True"
    type: ReconcileCompleted
Comment 4 Omri Hochman 2021-07-08 14:15:49 UTC 
if you can please reproduce with the latest ai-operator from ACM ?
Comment 5 Alexander Chuzhoy 2021-07-09 17:56:13 UTC 
Didn't reproduce with:

2.3.0-DOWNSTREAM-2021-07-09-09-53-06
4.8.0-rc.3

oc get pod -n rhacm                                          |grep assi
assisted-service-558fc94c5-4gmgk                                  2/2     Running   1          5m40s




oc logs -n rhacm                                              assisted-service-558fc94c5-4gmgk -c assisted-service
false
time="2021-07-09T17:49:31Z" level=info msg="Setting log format: text" func=main.InitLogs file="/remote-source/assisted-service/app/cmd/main.go:149"
time="2021-07-09T17:49:31Z" level=info msg="Setting Log Level: info" func=main.InitLogs file="/remote-source/assisted-service/app/cmd/main.go:155"
time="2021-07-09T17:49:31Z" level=info msg="Starting bm service" func=main.main file="/remote-source/assisted-service/app/cmd/main.go:183"
time="2021-07-09T17:49:31Z" level=info msg="Started service with OCP versions {\"4.6\":{\"display_name\":\"4.6\",\"release_image\":null,\"release_version\":null,\"rhcos_image\":\"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.3/rhcos-live.x86_64.iso\",\"rhcos_rootfs\":\"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.3/rhcos-live-rootfs.x86_64.img\",\"rhcos_version\":\"48.84.202107040900-0\",\"support_level\":null},\"4.7\":{\"display_name\":\"4.7\",\"release_image\":null,\"release_version\":null,\"rhcos_image\":\"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.3/rhcos-live.x86_64.iso\",\"rhcos_rootfs\":\"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.3/rhcos-live-rootfs.x86_64.img\",\"rhcos_version\":\"48.84.202107040900-0\",\"support_level\":null},\"4.8\":{\"display_name\":\"4.8\",\"release_image\":null,\"release_version\":null,\"rhcos_image\":\"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.3/rhcos-live.x86_64.iso\",\"rhcos_rootfs\":\"https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/4.8.0-rc.3/rhcos-live-rootfs.x86_64.img\",\"rhcos_version\":\"48.84.202107040900-0\",\"support_level\":null}}" func=main.main file="/remote-source/assisted-service/app/cmd/main.go:194"
time="2021-07-09T17:49:31Z" level=info msg="Connecting to DB" func=main.setupDB file="/remote-source/assisted-service/app/cmd/main.go:553"
time="2021-07-09T17:49:31Z" level=info msg="Failed to connect to DB, retrying" func=main.setupDB.func1 file="/remote-source/assisted-service/app/cmd/main.go:557" error="dial tcp [::1]:5432: connect: connection refused"
time="2021-07-09T17:49:33Z" level=info msg="Connected to DB" func=main.setupDB file="/remote-source/assisted-service/app/cmd/main.go:568"
I0709 17:49:34.849429       1 request.go:655] Throttling request took 1.041635676s, request: GET:https://[fd02::1]:443/apis/policy/v1beta1?timeout=32s
time="2021-07-09T17:49:38Z" level=info msg="Created local authenticator" func=github.com/openshift/assisted-service/pkg/auth.NewAuthenticator file="/remote-source/assisted-service/app/pkg/auth/authenticator.go:52" pkg=auth
time="2021-07-09T17:49:38Z" level=info msg="Configuring CNV Operator plugin" func=github.com/openshift/assisted-service/internal/operators/cnv.NewCNVOperator file="/remote-source/assisted-service/app/internal/operators/cnv/cnv_operator.go:43" config="{map[10de:1db6:true 10de:1eb8:true] map[15b3:1013:true 15b3:1015:true 15b3:1017:true 15b3:101b:true 8086:158b:true] true}"
time="2021-07-09T17:49:38Z" level=info msg="DeployTarget: k8s" func=main.main file="/remote-source/assisted-service/app/cmd/main.go:268"
W0709 17:49:38.453541       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-07-09T17:49:38Z" level=info msg="Attempting to acquire leader lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:120" configMap=assisted-service-leader-election-helper pkg=monitor-runner
W0709 17:49:38.457556       1 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2021-07-09T17:49:38Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-leader-election-helper pkg=monitor-runner
I0709 17:49:38.457688       1 leaderelection.go:243] attempting to acquire leader lease rhacm/assisted-service-leader-election-helper...
time="2021-07-09T17:49:38Z" level=info msg="Attempting to acquire leader lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:120" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-07-09T17:49:38Z" level=info msg="Start waiting for leader" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).waitForLeader" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:87" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-07-09T17:49:38Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-migration-helper pkg=migrationLeader
I0709 17:49:38.463960       1 leaderelection.go:243] attempting to acquire leader lease rhacm/assisted-service-migration-helper...
I0709 17:49:38.464165       1 leaderelection.go:253] successfully acquired lease rhacm/assisted-service-leader-election-helper
time="2021-07-09T17:49:38Z" level=info msg="Successfully acquired leadership lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:176" configMap=assisted-service-leader-election-helper pkg=monitor-runner
I0709 17:49:38.467532       1 leaderelection.go:253] successfully acquired lease rhacm/assisted-service-migration-helper
time="2021-07-09T17:49:38Z" level=info msg="Successfully acquired leadership lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:176" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-07-09T17:49:40Z" level=info msg="Got leader, stop waiting" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).waitForLeader" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:94" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-07-09T17:49:40Z" level=info msg="Start automigration" func=main.autoMigrationWithLeader.func1 file="/remote-source/assisted-service/app/cmd/main.go:670"
time="2021-07-09T17:49:40Z" level=info msg="Finished automigration" func=main.autoMigrationWithLeader.func1 file="/remote-source/assisted-service/app/cmd/main.go:676"
time="2021-07-09T17:49:40Z" level=info msg="Starting manual migrations" func=main.autoMigrationWithLeader.func1 file="/remote-source/assisted-service/app/cmd/main.go:678"
time="2021-07-09T17:49:40Z" level=info msg="Finished manual migrations" func=main.autoMigrationWithLeader.func1 file="/remote-source/assisted-service/app/cmd/main.go:684"
time="2021-07-09T17:49:40Z" level=info msg="Started Cluster State Monitor" func="github.com/openshift/assisted-service/pkg/thread.(*Thread).Start" file="/remote-source/assisted-service/app/pkg/thread/thread.go:40" pkg=cluster-monitor
time="2021-07-09T17:49:40Z" level=info msg="Started Host State Monitor" func="github.com/openshift/assisted-service/pkg/thread.(*Thread).Start" file="/remote-source/assisted-service/app/pkg/thread/thread.go:40" pkg=host-monitor
time="2021-07-09T17:49:40Z" level=info msg="Started Deletion Worker" func="github.com/openshift/assisted-service/pkg/thread.(*Thread).Start" file="/remote-source/assisted-service/app/pkg/thread/thread.go:40" garbagecollector="Deletion Worker"
time="2021-07-09T17:49:40Z" level=info msg="Started Image Expiration Monitor" func="github.com/openshift/assisted-service/pkg/thread.(*Thread).Start" file="/remote-source/assisted-service/app/pkg/thread/thread.go:40" pkg=image-expiration-monitor
time="2021-07-09T17:49:40Z" level=info msg="NO LONGER LEADER" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func2" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:180" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-07-09T17:49:40Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-07-09T17:49:40Z" level=info msg="Given context was cancelled, exiting leader elector" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:128" configMap=assisted-service-migration-helper pkg=migrationLeader
time="2021-07-09T17:49:40Z" level=info msg="waiting for REST api readiness before starting controllers" func=main.main.func4 file="/remote-source/assisted-service/app/cmd/main.go:489"
time="2021-07-09T17:49:40Z" level=info msg="Attempting to acquire leader lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:120" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-07-09T17:49:40Z" level=info msg="Start waiting for leader" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).waitForLeader" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:87" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-07-09T17:49:40Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
I0709 17:49:40.629692       1 leaderelection.go:243] attempting to acquire leader lease rhacm/assisted-service-baseiso-helper...
I0709 17:49:40.635348       1 leaderelection.go:253] successfully acquired lease rhacm/assisted-service-baseiso-helper
time="2021-07-09T17:49:40Z" level=info msg="Successfully acquired leadership lease" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:176" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-07-09T17:49:42Z" level=info msg="Got leader, stop waiting" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).waitForLeader" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:94" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-07-09T17:49:42Z" level=warning msg="ISO template version metadata not found, uploading latest" func=github.com/openshift/assisted-service/pkg/s3wrapper.HaveLatestMinimalTemplate file="/remote-source/assisted-service/app/pkg/s3wrapper/util.go:120" error="object minimal_templates_version.json was not found"
time="2021-07-09T17:50:19Z" level=info msg="Successfully uploaded file rhcos-48.84.202107040900-0.iso" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadStream" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:159" go-id=736 request_id=main-uploadISOs
time="2021-07-09T17:50:19Z" level=info msg="Successfully uploaded object rhcos-48.84.202107040900-0.iso" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadISOs" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:365" go-id=736 request_id=main-uploadISOs
time="2021-07-09T17:50:19Z" level=info msg="Extracting rhcos ISO (/data/rhcos-48.84.202107040900-0.iso)" func=github.com/openshift/assisted-service/pkg/s3wrapper.CreateAndUploadMinimalIso file="/remote-source/assisted-service/app/pkg/s3wrapper/util.go:93" go-id=736 request_id=main-uploadISOs
time="2021-07-09T17:50:20Z" level=info msg="Successfully uploaded file rhcos-48.84.202107040900-0.iso" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadStream" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:159" go-id=735 request_id=main-uploadISOs
time="2021-07-09T17:50:20Z" level=info msg="Successfully uploaded object rhcos-48.84.202107040900-0.iso" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadISOs" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:365" go-id=735 request_id=main-uploadISOs
time="2021-07-09T17:50:20Z" level=info msg="Extracting rhcos ISO (/data/rhcos-48.84.202107040900-0.iso)" func=github.com/openshift/assisted-service/pkg/s3wrapper.CreateAndUploadMinimalIso file="/remote-source/assisted-service/app/pkg/s3wrapper/util.go:93" go-id=735 request_id=main-uploadISOs
time="2021-07-09T17:50:21Z" level=info msg="Creating minimal ISO template" func="github.com/openshift/assisted-service/internal/isoeditor.(*rhcosEditor).CreateMinimalISOTemplate" file="/remote-source/assisted-service/app/internal/isoeditor/rhcos.go:89" go-id=736 request_id=main-uploadISOs
time="2021-07-09T17:50:21Z" level=info msg="Uploading minimal ISO (/data/isoeditor563614364)" func=github.com/openshift/assisted-service/pkg/s3wrapper.CreateAndUploadMinimalIso file="/remote-source/assisted-service/app/pkg/s3wrapper/util.go:107" go-id=736 request_id=main-uploadISOs
time="2021-07-09T17:50:21Z" level=info msg="Successfully uploaded file rhcos-48.84.202107040900-0-minimal.iso" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadStream" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:159" go-id=736 request_id=main-uploadISOs
time="2021-07-09T17:50:21Z" level=info msg="Successfully uploaded file minimal_templates_version.json" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadStream" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:159" go-id=736 request_id=main-uploadISOs
time="2021-07-09T17:50:22Z" level=info msg="Creating minimal ISO template" func="github.com/openshift/assisted-service/internal/isoeditor.(*rhcosEditor).CreateMinimalISOTemplate" file="/remote-source/assisted-service/app/internal/isoeditor/rhcos.go:89" go-id=735 request_id=main-uploadISOs
time="2021-07-09T17:50:22Z" level=info msg="Uploading minimal ISO (/data/isoeditor846563257)" func=github.com/openshift/assisted-service/pkg/s3wrapper.CreateAndUploadMinimalIso file="/remote-source/assisted-service/app/pkg/s3wrapper/util.go:107" go-id=735 request_id=main-uploadISOs
time="2021-07-09T17:50:22Z" level=info msg="Successfully uploaded file rhcos-48.84.202107040900-0-minimal.iso" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadStream" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:159" go-id=735 request_id=main-uploadISOs
time="2021-07-09T17:50:22Z" level=info msg="Successfully uploaded file minimal_templates_version.json" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadStream" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:159" go-id=735 request_id=main-uploadISOs
time="2021-07-09T17:50:24Z" level=info msg="Successfully uploaded file rhcos-48.84.202107040900-0.iso" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadStream" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:159" go-id=734 request_id=main-uploadISOs
time="2021-07-09T17:50:24Z" level=info msg="Successfully uploaded object rhcos-48.84.202107040900-0.iso" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadISOs" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:365" go-id=734 request_id=main-uploadISOs
time="2021-07-09T17:50:24Z" level=info msg="Extracting rhcos ISO (/data/rhcos-48.84.202107040900-0.iso)" func=github.com/openshift/assisted-service/pkg/s3wrapper.CreateAndUploadMinimalIso file="/remote-source/assisted-service/app/pkg/s3wrapper/util.go:93" go-id=734 request_id=main-uploadISOs
time="2021-07-09T17:50:26Z" level=info msg="Creating minimal ISO template" func="github.com/openshift/assisted-service/internal/isoeditor.(*rhcosEditor).CreateMinimalISOTemplate" file="/remote-source/assisted-service/app/internal/isoeditor/rhcos.go:89" go-id=734 request_id=main-uploadISOs
time="2021-07-09T17:50:26Z" level=info msg="Uploading minimal ISO (/data/isoeditor851399809)" func=github.com/openshift/assisted-service/pkg/s3wrapper.CreateAndUploadMinimalIso file="/remote-source/assisted-service/app/pkg/s3wrapper/util.go:107" go-id=734 request_id=main-uploadISOs
time="2021-07-09T17:50:26Z" level=info msg="Successfully uploaded file rhcos-48.84.202107040900-0-minimal.iso" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadStream" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:159" go-id=734 request_id=main-uploadISOs
time="2021-07-09T17:50:26Z" level=info msg="Successfully uploaded file minimal_templates_version.json" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).UploadStream" file="/remote-source/assisted-service/app/pkg/s3wrapper/filesystem.go:159" go-id=734 request_id=main-uploadISOs
time="2021-07-09T17:50:26Z" level=info msg="API is enabled" func="main.(*ApiEnabler).Enable" file="/remote-source/assisted-service/app/cmd/main.go:659"
time="2021-07-09T17:50:26Z" level=info msg="NO LONGER LEADER" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).createLeaderElector.func2" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:180" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-07-09T17:50:26Z" level=info msg="Starting leader elections process" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:126" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-07-09T17:50:26Z" level=info msg="Given context was cancelled, exiting leader elector" func="github.com/openshift/assisted-service/pkg/leader.(*Elector).StartLeaderElection.func1" file="/remote-source/assisted-service/app/pkg/leader/leaderelector.go:128" configMap=assisted-service-baseiso-helper pkg=baseISOUploadLeader
time="2021-07-09T17:50:27Z" level=info msg="REST api is now ready, starting controllers" func=main.main.func4 file="/remote-source/assisted-service/app/cmd/main.go:492"
I0709 17:50:27.638356       1 leaderelection.go:243] attempting to acquire leader lease rhacm/77190dcb.agent-install.openshift.io...
E0709 17:50:27.650115       1 event.go:329] Could not construct reference to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"77190dcb.agent-install.openshift.io", GenerateName:"", Namespace:"rhacm", SelfLink:"", UID:"710e95b8-e6f0-4821-832d-46bba2323b88", ResourceVersion:"606914", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63761449866, loc:(*time.Location)(0x3a84b20)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"assisted-service", Operation:"Update", APIVersion:"coordination.k8s.io/v1", Time:(*v1.Time)(0xc00111d2d8), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00111d2f0)}}}, Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil), LeaseDurationSeconds:(*int32)(nil), AcquireTime:(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil), LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type v1.Lease in scheme "pkg/runtime/scheme.go:100"'. Will not report event: 'Normal' 'LeaderElection' 'assisted-service-558fc94c5-4gmgk_36bd9d83-7ea0-41e7-8f05-4cdbb0ce58ed became leader'
I0709 17:50:27.650186       1 leaderelection.go:253] successfully acquired lease rhacm/77190dcb.agent-install.openshift.io
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-master-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=b4f90af3-eba7-4e93-a2a6-a76773806d90
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-master-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=b4f90af3-eba7-4e93-a2a6-a76773806d90
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-master-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=b4f90af3-eba7-4e93-a2a6-a76773806d90
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-master-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=61237ff5-5fb1-48b9-9efe-5e411d4275af
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-master-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=61237ff5-5fb1-48b9-9efe-5e411d4275af
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-master-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=61237ff5-5fb1-48b9-9efe-5e411d4275af
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-master-2 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=2584af03-44ee-4058-b6b6-d955de17a0f9
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-master-2 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=2584af03-44ee-4058-b6b6-d955de17a0f9
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-master-2 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=2584af03-44ee-4058-b6b6-d955de17a0f9
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-worker-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=081fb0f6-55f7-4dff-a72f-7b28a71bac13
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-worker-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=081fb0f6-55f7-4dff-a72f-7b28a71bac13
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-worker-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=081fb0f6-55f7-4dff-a72f-7b28a71bac13
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-worker-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=02f227b5-88ee-4444-bbc1-bcd6a64dc646
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-worker-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=02f227b5-88ee-4444-bbc1-bcd6a64dc646
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-worker-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=02f227b5-88ee-4444-bbc1-bcd6a64dc646
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-master-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=a88cd7de-7c93-43ee-975c-7b566a6d6d09
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-master-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=a88cd7de-7c93-43ee-975c-7b566a6d6d09
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-master-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=a88cd7de-7c93-43ee-975c-7b566a6d6d09
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-master-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=1815b4f5-b984-4116-b295-64ce2dc300f7
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-master-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=1815b4f5-b984-4116-b295-64ce2dc300f7
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-master-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=1815b4f5-b984-4116-b295-64ce2dc300f7
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-master-2 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=d21a9929-a929-42ae-bcd2-0ff04d611e97
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-master-2 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=d21a9929-a929-42ae-bcd2-0ff04d611e97
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-master-2 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=d21a9929-a929-42ae-bcd2-0ff04d611e97
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-worker-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=f47c272f-69d6-4bdc-8037-5bbb455c3a1f
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-worker-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=f47c272f-69d6-4bdc-8037-5bbb455c3a1f
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-worker-0 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=f47c272f-69d6-4bdc-8037-5bbb455c3a1f
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile started" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:164" bare_metal_host=openshift-worker-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=0e353773-c9a3-4f65-b4bd-c3bf87ac0faa
time="2021-07-09T17:50:28Z" level=info msg="Stopping reconcileBMH: BMH does not have an InfraEnv" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).reconcileBMH" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:595" bare_metal_host=openshift-worker-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=0e353773-c9a3-4f65-b4bd-c3bf87ac0faa
time="2021-07-09T17:50:28Z" level=info msg="BareMetalHost Reconcile ended" func="github.com/openshift/assisted-service/internal/controller/controllers.(*BMACReconciler).Reconcile.func1" file="/remote-source/assisted-service/app/internal/controller/controllers/bmh_agent_controller.go:161" bare_metal_host=openshift-worker-1 bare_metal_host_namespace=openshift-machine-api go-id=1169 request_id=0e353773-c9a3-4f65-b4bd-c3bf87ac0faa
Comment 9 errata-xmlrpc 2021-10-12 06:01:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.8.14 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3682