Bug 1866543
| Summary: | Spoke cluster creation in a disconnected env: Cluster is stuck in "Cluster is pending import" | ||
|---|---|---|---|
| Product: | Red Hat Advanced Cluster Management for Kubernetes | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | Cluster Lifecycle | Assignee: | Hao Liu <haoli> |
| Status: | CLOSED ERRATA | QA Contact: | magchen@redthat.com <magchen> |
| Severity: | urgent | Docs Contact: | Christopher Dawson <cdawson> |
| Priority: | unspecified | ||
| Version: | rhacm-2.0 | CC: | gghezzo, yjoseph |
| Target Milestone: | --- | Flags: | sasha:
rhacm-2.0.z?
gghezzo: rhacm-2.1+ |
| Target Release: | rhacm-2.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | rhacm-2.1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-11-05 11:55:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Alexander Chuzhoy
2020-08-05 20:36:32 UTC
This is related to an image not being reachable:
The following command was executed against the spoke cluster:
[kni@provisionhost-0-0 ~]$ oc describe pod -n open-cluster-management-agent klusterlet-8fc468666-q94xf
Name: klusterlet-8fc468666-q94xf
Namespace: open-cluster-management-agent
Priority: 0
Node: worker-1-1/192.168.124.135
Start Time: Wed, 05 Aug 2020 19:54:18 +0000
Labels: app=klusterlet
pod-template-hash=8fc468666
Annotations: k8s.ovn.org/pod-networks:
{"default":{"ip_addresses":["10.128.2.6/23"],"mac_address":"1a:f8:16:80:02:07","gateway_ips":["10.128.2.1"],"ip_address":"10.128.2.6/23","...
k8s.v1.cni.cncf.io/network-status:
[{
"name": "ovn-kubernetes",
"interface": "eth0",
"ips": [
"10.128.2.6"
],
"mac": "1a:f8:16:80:02:07",
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "ovn-kubernetes",
"interface": "eth0",
"ips": [
"10.128.2.6"
],
"mac": "1a:f8:16:80:02:07",
"default": true,
"dns": {}
}]
openshift.io/scc: restricted
Status: Pending
IP: 10.128.2.6
IPs:
IP: 10.128.2.6
Controlled By: ReplicaSet/klusterlet-8fc468666
Containers:
klusterlet:
Container ID:
Image: registry.redhat.io/rhacm2/registration-rhel8-operator@sha256:0630bca8263f93a4a1348e6bd5a8689157739e8ab09d4f3354acb4be1bf66dda
Image ID:
Port: <none>
Host Port: <none>
Args:
/registration-operator
klusterlet
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Liveness: http-get https://:8443/healthz delay=2s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:8443/healthz delay=2s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from klusterlet-token-t2f5p (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
klusterlet-token-t2f5p:
Type: Secret (a volume populated by a Secret)
SecretName: klusterlet-token-t2f5p
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned open-cluster-management-agent/klusterlet-8fc468666-q94xf to worker-1-1
Normal AddedInterface 90m multus Add eth0 [10.128.2.6/23]
Warning Failed 84m (x4 over 89m) kubelet, worker-1-1 Failed to pull image "registry.redhat.io/rhacm2/registration-rhel8-operator@sha256:0630bca8263f93a4a1348e6bd5a8689157739e8ab09d4f3354acb4be1bf66dda": rpc error: code = Unknown desc = error pinging docker registry registry.redhat.io: Get https://registry.redhat.io/v2/: dial tcp 104.126.247.209:443: i/o timeout
Normal BackOff 20m (x240 over 89m) kubelet, worker-1-1 Back-off pulling image "registry.redhat.io/rhacm2/registration-rhel8-operator@sha256:0630bca8263f93a4a1348e6bd5a8689157739e8ab09d4f3354acb4be1bf66dda"
Warning Failed 9m59s (x275 over 89m) kubelet, worker-1-1 Error: ImagePullBackOff
Warning Failed 5m7s (x18 over 89m) kubelet, worker-1-1 Error: ErrImagePull
Normal Pulling 3s (x19 over 90m) kubelet, worker-1-1 Pulling image "registry.redhat.io/rhacm2/registration-rhel8-operator@sha256:0630bca8263f93a4a1348e6bd5a8689157739e8ab09d4f3354acb4be1bf66dda"
In order to make the import complete successful, had to: 1. mirror the following images to the local registry: registry.redhat.io/rhacm2/registration-rhel8-operator registry.redhat.io/rhacm2/work-rhel8 registry.redhat.io/rhacm2/registration-rhel8 registry.redhat.io/rhacm2/endpoint-component-rhel8-operator 2. Create an imagecontentsourcepolicy for registry.redhat.io/rhacm2 pointing to the local mirror. G2Bsync 670198597 comment TheRealHaoLiu Thu, 06 Aug 2020 21:22:14 UTC G2Bsync `ImagePullBackOff` `Get https://registry.redhat.io/v2/: dial tcp 104.126.247.209:443: i/o timeout` is this cluster disconnected? if yes, does this cluster get created with the correct ImageContentSource? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Advanced Cluster Management for Kubernetes version 2.1 images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:4954 |