Bug 1710333

Summary:	kubevirt-node-labeller and virt-template-validator have problem to start
Product:	Container Native Virtualization (CNV)	Reporter:	Lukas Bednar <lbednar>
Component:	Installation	Assignee:	Francesco Romani <fromani>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Irina Gulina <igulina>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.0	CC:	cnv-qe-bugs, fdeutsch, fromani, gouyang, grajaiya, pep, rhallise, sgordon
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	2.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-07-29 07:24:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Lukas Bednar 2019-05-15 11:02:25 UTC

Description of problem:

(In reply to David Zager from comment #6)
> Denys Shchedrivyi mentioned the following issue with the
> kubevirt-node-labeller on the cnv-devel list (Subj: [cnv]
> hco-bundle-registry:v2.0.0-13) that I believe comes from the
> kubevirt-ssp-operator: 
> 
> [root@dell-r640-010 ~]# oc describe pod -n kubevirt-hyperconverged 
> kubevirt-node-labeller-hzwxl
> .
>   Warning  Failed            1h (x4 over 1h)     kubelet,
> working-jww4k-worker-0-6dqp9  Failed to pull image
> "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-
> virtualization/kvm-info-nfd-plugin:v0.4.0": rpc error: code = Unknown desc =
> Error reading manifest v0.4.0 in
> brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-
> virtualization/kvm-info-nfd-plugin: unknown: Not Found
>   Warning  Failed            1h (x4 over 1h)     kubelet,
> working-jww4k-worker-0-6dqp9  Error: ErrImagePull
>   Warning  Failed            5m (x331 over 1h)   kubelet,
> working-jww4k-worker-0-6dqp9  Error: ImagePullBackOff
>   Normal   BackOff           44s (x353 over 1h)  kubelet,
> working-jww4k-worker-0-6dqp9  Back-off pulling image
> "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-
> virtualization/kvm-info-nfd-plugin:v0.4.0"


Version-Release number of selected component (if applicable):
hco-2.0.0-11

How reproducible:


Steps to Reproduce:
1. Deploy CNV on top of OCP
2.
3.

Actual results: labeler should come up


Expected results: can not pull image of kvm-info-nfd-plugin


Additional info:
From Denys's mail
[root@dell-r640-010 ~]# oc get pod -n kubevirt-hyperconverged 
NAME                                               READY     STATUS                  RESTARTS   AGE
cdi-operator-c78d55bd7-nftsp                       1/1       Running                 0          68m
cluster-network-addons-operator-66655bd69b-xsvtt   1/1       Running                 0          68m
hco-operator-5d875c67b8-w7qqz                      1/1       Running                 0          68m
kubevirt-node-labeller-hzwxl                       0/1       Init:ImagePullBackOff   0          65m
kubevirt-node-labeller-nspgj                       0/1       Init:ImagePullBackOff   0          65m
kubevirt-ssp-operator-578cb75cbd-w84fv             1/1       Running                 0          68m
kubevirt-web-ui-operator-6db98f677f-c2zd4          1/1       Running                 0          68m
node-maintenance-operator-5b489b5556-dnf8t         1/1       Running                 0          68m
virt-api-5d9f66dc76-qk64k                          1/1       Running                 0          66m
virt-api-5d9f66dc76-x95wk                          1/1       Running                 0          66m
virt-controller-6f8746f697-95tcp                   1/1       Running                 0          65m
virt-controller-6f8746f697-m8tw4                   1/1       Running                 0          65m
virt-handler-5rwnn                                 1/1       Running                 0          65m
virt-handler-cjxgt                                 1/1       Running                 0          65m
virt-operator-7f59d8df84-9jg9w                     1/1       Running                 0          68m
virt-operator-7f59d8df84-p6kt7                     1/1       Running                 0          68m
virt-template-validator-8dcd6dccc-96mnt            0/1       ImagePullBackOff        0          65m


[root@dell-r640-010 ~]# oc describe pod -n kubevirt-hyperconverged  kubevirt-node-labeller-hzwxl
.
  Warning  Failed            1h (x4 over 1h)     kubelet, working-jww4k-worker-0-6dqp9  Failed to pull image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kvm-info-nfd-plugin:v0.4.0": rpc error: code = Unknown desc = Error reading manifest v0.4.0 in brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kvm-info-nfd-plugin: unknown: Not Found
  Warning  Failed            1h (x4 over 1h)     kubelet, working-jww4k-worker-0-6dqp9  Error: ErrImagePull
  Warning  Failed            5m (x331 over 1h)   kubelet, working-jww4k-worker-0-6dqp9  Error: ImagePullBackOff
  Normal   BackOff           44s (x353 over 1h)  kubelet, working-jww4k-worker-0-6dqp9  Back-off pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kvm-info-nfd-plugin:v0.4.0"


[root@dell-r640-010 ~]# oc describe pod -n kubevirt-hyperconverged  virt-template-validator-8dcd6dccc-96mnt
.
  Warning  Failed       1h (x4 over 1h)     kubelet, working-jww4k-worker-0-z6t7b  Failed to pull image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-template-validator:v0.4.8": rpc error: code = Unknown desc = Error reading manifest v0.4.8 in brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-template-validator: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /pulp/docker/v2/redhat-container-native-virtualization-kubevirt-template-validator/manifests/1/v0.4.8 was not found on this server.</p>\n</body></html>\n"
  Warning  Failed       1h (x4 over 1h)     kubelet, working-jww4k-worker-0-z6t7b  Error: ErrImagePull
  Warning  Failed       33m (x217 over 1h)  kubelet, working-jww4k-worker-0-z6t7b  Error: ImagePullBackOff
  Normal   BackOff      3m (x350 over 1h)   kubelet, working-jww4k-worker-0-z6t7b  Back-off pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-template-validator:v0.4.8"

Comment 1 Francesco Romani 2019-05-15 11:25:31 UTC

Will be fixed by new kubevirt-ssp-operator build and https://github.com/kubevirt/hyperconverged-cluster-operator/pull/94

Comment 3 Guohua Ouyang 2019-05-16 06:30:22 UTC

Add testBlocker because it prevent us to create VM:
# oc create -f vm-cirros.yaml 
Error from server (InternalError): error when creating "vm-cirros.yaml": Internal error occurred: failed calling admission webhook "virt-template-admission.kubevirt.io": Post https://virt-template-validator.kubevirt-hyperconverged.svc:443/virtualmachine-template-validate?timeout=30s: no endpoints available for service "virt-template-validator"

Comment 5 Gowrishankar Rajaiyan 2019-05-16 19:09:25 UTC

Looks good with hco-bundle-registry:v2.0.0-14

$ oc get pods -n kubevirt-hyperconverged
NAME                                              READY     STATUS    RESTARTS   AGE
cdi-operator-5cf5f7df79-g7l75                     1/1       Running   0          11m
cluster-network-addons-operator-5fc94fc94-bkczt   1/1       Running   0          11m
hco-operator-64c9bc495d-gkvft                     1/1       Running   0          11m
kubevirt-node-labeller-hs6vm                      1/1       Running   0          8m48s
kubevirt-node-labeller-nb5jw                      1/1       Running   0          8m48s
kubevirt-ssp-operator-76bd5b68c-p9fw4             1/1       Running   0          11m
kubevirt-web-ui-operator-5599867479-x5ndq         1/1       Running   0          11m
node-maintenance-operator-7c4bd6546d-69qzl        1/1       Running   0          11m
virt-api-5759c89548-4qrxh                         1/1       Running   0          8m43s
virt-api-5759c89548-bkns2                         1/1       Running   0          8m43s
virt-controller-75d85dc4d-mgtp5                   1/1       Running   0          7m13s
virt-controller-75d85dc4d-ncm2s                   1/1       Running   0          7m13s
virt-handler-jfgsc                                1/1       Running   1          7m13s
virt-handler-n2l9f                                1/1       Running   0          7m13s
virt-operator-74698f4dcd-27cjn                    1/1       Running   0          11m
virt-operator-74698f4dcd-dvspk                    1/1       Running   0          11m
virt-template-validator-85d7964c78-bgvlf          1/1       Running   0          8m48s

Comment 6 Ryan Hallisey 2019-06-03 12:01:32 UTC

Has this been verified?

Comment 7 Irina Gulina 2019-06-03 12:09:12 UTC

Verfied, see BZ 1709677#c12