Bug 1959982

Summary:	SNO disconnected ipv6 deployment failed generating install config for cluster
Product:	OpenShift Container Platform	Reporter:	bjacot
Component:	assisted-installer	Assignee:	Mat Kowalski <mko>
assisted-installer sub component:	Deployment Operator	QA Contact:	bjacot
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	urgent
Priority:	urgent	CC:	aos-bugs, asegurap, ccrum, fpercoco, keyoung, mcornea, mfilanov, sasha
Version:	4.8	Keywords:	TestBlocker, Triaged
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	AI-Team-Platform
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-05-20 13:25:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description bjacot 2021-05-12 18:34:27 UTC

during SNO ipv6 disconnected deployment i am seeing an error in the assisted-service pod.  I am not seeing the SNO deployment progressing.  It appears AI is sourceing an ICSP.  


time="2021-05-12T18:25:26Z" level=error msg="level=fatal msg=failed to fetch Master Machines: failed to load asset \"Install Config\": invalid \"install-config.yaml\" file: [imageContentSources[1].source: Invalid value: \"registry-proxy.engineering.redhat.com\": failed to parse: repository name must be canonical, imageContentSources[1].mirrors[0]: Invalid value: \"registry.ocp-edge-cluster-bjacot-0.qe.lab.redhat.com:5000\"

Then: Failing to prepare install config
time="2021-05-12T18:25:26Z" level=warning msg="Failed to prepare installation of cluster 3302d565-5287-413d-be92-324028d09ee0" func="github.com/openshift/assisted-service/internal/cluster.(*Manager).HandlePreInstallError" file="/go/src/github.com/openshift/origin/internal/cluster/cluster.go:707" error="failed generating install config for cluster 3302d565-5287-413d-be92-324028d09ee0: exit status 1" go-id=177477 pkg=cluster-state request_id=

[kni@provisionhost-0-0 ~]$ oc get clusterdeployments.hive.openshift.io -n assisted-installer -o=custom-columns='STATUS:status.conditions[-1].message'
STATUS
The cluster requirements are met
[kni@provisionhost-0-0 ~]$ oc get clusterdeployments.hive.openshift.io -n assisted-installer -o=custom-columns='STATUS:status.conditions[-0].message'
STATUS
Condition Initialized


From the SNO node:
May 12 17:17:20 sno.ocp-edge-cluster-bjacot-0.qe.lab.redhat.com podman[1521]: Copying config sha256:c17a88c4ac42a8cf39e3eec2dd22730f1f77bbc723b6fe62df1fcce8a8e7fc54
May 12 17:17:20 sno.ocp-edge-cluster-bjacot-0.qe.lab.redhat.com podman[1521]: Writing manifest to image destination
May 12 17:17:20 sno.ocp-edge-cluster-bjacot-0.qe.lab.redhat.com podman[1521]: Storing signatures
May 12 17:17:37 sno.ocp-edge-cluster-bjacot-0.qe.lab.redhat.com systemd[1]: agent.service: Found left-over process 1911 (conmon) in control group while starting unit. Ignoring.
May 12 17:17:37 sno.ocp-edge-cluster-bjacot-0.qe.lab.redhat.com systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 12 17:17:37 sno.ocp-edge-cluster-bjacot-0.qe.lab.redhat.com systemd[1]: agent.service: Found left-over process 1917 (podman) in control group while starting unit. Ignoring.
May 12 17:17:37 sno.ocp-edge-cluster-bjacot-0.qe.lab.redhat.com systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
May 12 17:17:37 sno.ocp-edge-cluster-bjacot-0.qe.lab.redhat.com systemd[1]: Started agent.service.

Comment 1 bjacot 2021-05-13 11:11:23 UTC

See an error with creating manifests

time="2021-05-13T10:44:39Z" level=error msg="error running openshift-install create manifests" func="github.com/openshift/assisted-service/internal/ignition.(*installerGenerator).runCreateCommand" file="/go/src/github.com/openshift/origin/internal/ignition/ignition.go:1185" go-id=2016182 pkg=local-job-wrapper request_id=

time="2021-05-13T10:52:53Z" level=info msg="Creating manifest in cluster 677d1040-fba6-4669-95ac-fafbfcb31ca6" func="github.com/openshift/assisted-service/internal/manifests.(*Manifests).CreateClusterManifestInternal" file="/go/src/github.com/openshift/origin/internal/manifests/manifests.go:69" go-id=706 pkg=manifests request_id=
time="2021-05-13T10:52:53Z" level=info msg="Successfully uploaded file 677d1040-fba6-4669-95ac-fafbfcb31ca6/manifests/openshift/dnsmasq-bootstrap-in-place.yaml" func="github.com/openshift/assisted-service/pkg/s3wrapper.(*FSClient).Upload" file="/go/src/github.com/openshift/origin/pkg/s3wrapper/filesystem.go:76" go-id=706 request_id=
time="2021-05-13T10:52:53Z" level=info msg="Done creating manifest openshift/dnsmasq-bootstrap-in-place.yaml for cluster 677d1040-fba6-4669-95ac-fafbfcb31ca6" func="github.com/openshift/assisted-service/internal/manifests.(*Manifests).CreateClusterManifestInternal" file="/go/src/github.com/openshift/origin/internal/manifests/manifests.go:112" go-id=706 pkg=manifests request_id=
time="2021-05-13T10:52:53Z" level=info msg="Successfully prepared cluster <677d1040-fba6-4669-95ac-fafbfcb31ca6> for installation" func="github.com/openshift/assisted-service/internal/bminventory.(*bareMetalInventory).InstallClusterInternal" file="/go/src/github.com/openshift/origin/internal/bminventory/inventory.go:1230" go-id=706 pkg=Inventory request_id=
time="2021-05-13T10:52:53Z" level=info msg="Selected network type OVNKubernetes for cluster 677d1040-fba6-4669-95ac-fafbfcb31ca6" func="github.com/openshift/assisted-service/internal/installcfg.(*installConfigBuilder).getBasicInstallConfig" file="/go/src/github.com/openshift/origin/internal/installcfg/installcfg.go:183" pkg=installcfg
time="2021-05-13T10:52:53Z" level=info msg="None-Platform: Selected bootstrap machine network CIDR fd2e:6f44:5dd8::/64 for cluster 677d1040-fba6-4669-95ac-fafbfcb31ca6" func="github.com/openshift/assisted-service/internal/installcfg.(*installConfigBuilder).getInstallConfig" file="/go/src/github.com/openshift/origin/internal/installcfg/installcfg.go:360" pkg=installcfg
time="2021-05-13T10:52:53Z" level=info msg="Reconcile has been called for ClusterDeployment name=sno-cluster-deployment namespace=assisted-installer" func="github.com/openshift/assisted-service/internal/controller/controllers.(*ClusterDeploymentsReconciler).Reconcile" file="/go/src/github.com/openshift/origin/internal/controller/controllers/clusterdeployments_controller.go:89"
time="2021-05-13T10:52:53Z" level=info msg="Reconcile has been called for InfraEnv name=sno-infraenv namespace=assisted-installer" func="github.com/openshift/assisted-service/internal/controller/controllers.(*InfraEnvReconciler).Reconcile" file="/go/src/github.com/openshift/origin/internal/controller/controllers/infraenv_controller.go:66"
time="2021-05-13T10:52:53Z" level=info msg="Reconcile has been called for ClusterDeployment name=sno-cluster-deployment namespace=assisted-installer" func="github.com/openshift/assisted-service/internal/controller/controllers.(*ClusterDeploymentsReconciler).Reconcile" file="/go/src/github.com/openshift/origin/internal/controller/controllers/clusterdeployments_controller.go:89"
time="2021-05-13T10:52:53Z" level=error msg="error running openshift-install create manifests" func="github.com/openshift/assisted-service/internal/ignition.(*installerGenerator).runCreateCommand" file="/go/src/github.com/openshift/origin/internal/ignition/ignition.go:1185" go-id=2036802 pkg=local-job-wrapper request_id=

Comment 2 Michael Filanov 2021-05-13 13:35:23 UTC

Can you please try to reproduce it with an API  https://github.com/openshift/assisted-service/blob/master/docs/operator.md#mirror-registry-configuration ?
I looks like there is a problem with the annotation and it may cause a conflict with the API. 
But maybe there is a problem with the format as well.

Comment 6 Mat Kowalski 2021-05-20 13:25:58 UTC


*** This bug has been marked as a duplicate of bug 1960378 ***